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I. 


INTRODUCTION 


Air  Force  Smart  Operations  21  (AFS021)  initiated  a  movement  to  improve 
combat  capability  to  include  improve  decision  making.  The  uncertainty  of  weather 
forecasts  contributes  to  mission  risk.  Combined  with  ORM  principles,  ensemble  data  can 
improve  combat  capability  through  better  decision  making  (Eckel  et  al.,  2007).  The  focus 
of  this  study  is  not  to  prove  the  value  of  ensemble  data.  Rather,  the  focus  is  to  transform 
ensemble  data  into  optimized  decision  input  and  communicate  it  an  effective  manner  so  it 
can  be  incorporated  into  the  decision  making  process.  ‘Optimized’  describes  decision 
input  that  incorporates  a  complete  picture  of  uncertainty  by  estimating  the  uncertainty  of 
forecast  probability  (ambiguity)  and  risk  tolerance  information.  Upper  level  turbulence 
data  will  serve  as  the  example  weather  parameter. 

The  three  main  objectives  are  to:  1)  Furnish  a  process  to  account  for  uncertainty 
in  both  the  ensemble  data  and  user  risk  tolerance  for  the  decision  input,  2)  create  an 
effective  visualization  to  for  varying  levels  of  risk  intolerance,  mission  thresholds  for 
turbulence,  flight  levels,  and  forecast  hours  that  is  educational  and  practical  for  the  user 
3)  demonstrate  potential  value  added  to  the  user  when  using  the  research  derived  decision 
input  product  through  a  comparison  of  current  decision  input  products. 

The  thesis  organized  into  three  main  chapters:  Introduction  (Chapter  I), 
Background  (Chapter  II),  Methodology  (Chapter  III),  Results  (Chapter  IV),  and 
Conclusions  (Chapter  V).  Chapter  III  fulfills  objective  one  through  a  description  of  a 
newly  fashioned  ambiguity  calculation  method,  called  “Taijitu”  (Eckel,  2008).  Chapter 
III  also  fulfills  objective  two  by  characterizing  the  process  to  create  the  visualizations  and 
display  it  using  a  graphical  using  interface.  Chapter  IV  fulfills  objective  three  with  two 
cases  which  compare  the  different  decision  input  products  using  scenarios  of  different 
risk  tolerances  and  forecast  hours. 

The  main  benefit  to  this  study  will  be  to  demonstrate  the  capability  of  stochastic 
model  data  incorporation  into  the  decision  making  process.  It  will  serve  as  an  example 
for  further  education  and  development  of  stochastic  methods  in  the  Air  Force  and 

Department  of  Defense  to  improve  combat  capability. 

1 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


2 


II.  BACKGROUND 


A  basic  understanding  of  meteorology  (with  emphasis  in  numerical  weather 
prediction),  decision  theory,  statistics,  and  military  weapon  systems  is  important  to 
understand  the  complex  problem  of  applying  ensemble  forecast  application  products  to 
DoD  operations.  In  this  chapter,  background  will  be  explicitly  given  on  ensemble 
forecasting,  weather  risk  management,  and  visualizations.  It  is  assumed  the  reader  will 
have  some  basic  understanding  of  statistics  and  military  weapon  systems. 

A.  ATMOSPHERIC  PREDICTION 

1.  Atmospheric  Uncertainty 

Prediction  of  the  atmosphere  is  inherently  uncertain.  Prediction  of  a  future  state 
of  a  dynamical  system,  such  as  the  atmosphere,  is  possible  using  the  system’s  rules  and 
current  state  (Eckel,  2007).  From  the  atmospheric  modeling  perspective,  the  system’s 
rules  are  the  set  of  equations  that  govern  atmospheric  motion  and  the  current  state  is  the 
set  of  initial  conditions  that  estimate  the  atmosphere’s  current  state.  Prediction  is  limited 
when  knowledge  of  one  or  both  is  erred  or  incomplete,  which  results  in  exponentially 
growing  forecast  errors  (Eckel,  2007).  This  uncertainty  of  prediction  stems  from  the 
chaotic  nature  of  the  atmosphere.  The  mathematical  definition  of  chaos  is  “a  dynamical 
system  that  has  a  sensitive  dependence  on  its  initial  conditions”  (American  Heritage 
Dictionary,  2004).  Edward  Lorenz  (1963  a,  b)  described  “dynamical  chaos”  as  the  time 
evolution  of  “a  nonlinear,  deterministic  dynamical  system,  which  is  extremely  sensitive 
to  the  initial  conditions  of  the  system”  (Wilks,  2006).  Error  in  this  initial  description  of 
the  atmosphere  is  the  primary  contributor  to  the  lack  of  atmospheric  predictability 
(Lorenz,  1963  a,  b). 
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Initial  conditions  (ICs)  are  a  set  of  values  for  the  dependant  variables  that  describe 
the  state  of  atmosphere  at  certain  point  in  time.  There  exists  two  sets  of  ICs.  The  first  set 
of  ICs  describes  the  true  state  of  the  atmosphere.  The  second  set  of  ICs  is  analyzed  from 
observation  of  the  atmosphere.  The  state  variables  measured  include  temperature,  dew 
point,  relative  humidity,  three  dimensional  wind  speed  components,  and  pressure.  The 
second  set  of  ICs  are  inherently  erred  because  the  atmosphere  is  never  completely  and 
accurately  observed.  The  data  network  is  incomplete  with  course  temporal  and  spatial 
resolution  and  the  observations  are  imprecise  and/or  erred.  Hence,  the  observed/analyzed 
ICs  will  never  be  the  same  as  the  true  ICs  (Wilks,  2006). 

Forecast  error  arises  when  the  observed  (incomplete/erred)  ICs  are  fed  into  an 
imperfect  numerical  weather  prediction  (NWP)  model.  A  NWP  model  is  composed  of  a 
set  of  differential  equations,  with  time  as  an  independent  variable,  which  govern  the 
atmosphere’s  motion.  It  integrates  these  atmospheric  equations  using  complex 
algorithms  to  simulate  the  atmospheric  behavior  over  time,  generating  a  forecast  with 
varying  degrees  of  accuracy  (Eckel,  2007).  The  model  is  unable  to  simulate  the  true 
behavior  of  the  atmosphere  because  of  inexact  numerical  and  computational  methods, 
boundary  conditions,  and  physics  within  the  model.  Errors  in  the  ICs  and  model  may  be 
small  (even  undetectable),  but  over  a  typical  forecast  period,  they  lead  to  a  cascade  of 
error  that  makes  the  forecast  diverge  exponentially  from  the  truth. 

Predictions  made  by  the  deficient  NWP  model,  which  is  initialized  using  the  erred 
ICs,  are  reasonably  accurate  in  the  short-term,  but  the  solutions  eventually  diverges  from 
truth  and  the  ability  to  forecast  accurately  is  lost  (Eckel,  2007).  Kalnay  (2003)  states 
even  though  two  ICs  may  be  almost  identical  initially,  an  unstable  dynamical  system, 
such  as  the  atmosphere,  will  cause  the  trajectories  to  drift  apart  (Figure  la).  Conversely, 
a  stable  dynamical  system  with  “stationary  or  periodic  orbits”  will  cause  two  ICs  initially 
on  different  trajectories  to  converge  together  (Figure  lb).  On  the  synoptic  scale,  the 
solutions  may  start  at  similar  point  and  be  close  for  two  to  three  days,  but  further  in  time, 
the  two  solutions  diverge  (Kalnay,  2003). 
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Figure  1 .  a)  Dynamical  System  with  instabilities  b)  Stable  Dynamical  System  with 

stationary  or  periodic  orbits  [From  Kalnay,  2003]. 


Lorenz  proved  this  concept  when  he  performed  an  “identical  twin”  experiment, 
where  he  compared  two  runs  of  the  same  model  but  with  slightly  different  ICs.  He  found 
that  the  two  runs’  solutions  diverged  dramatically  and  exhibited  totally  different 
trajectories  (Kalnay,  2003).  The  initial  round-off  errors  were  amplified  until  they 
dominated  the  solution  (Lorenz,  1993).  Lorenz  (1963  a,  b)  ascertained  the  atmosphere, 
similar  to  any  dynamical  system  with  instabilities,  has  a  ‘finite  limit  of  predictability’. 

Atmospheric  predictability  is  the  degree  to  which  a  phenomenon  can  be  foretold 
with  an  average  accuracy  greater  than  that  of  an  unskilled  reference  forecast  (Eckel, 
2007).  Generally,  prediction  decreases  with  increasing  lead-time.  When  predictability  is 
high,  the  theoretical  limit  for  forecasting  is  two  weeks  (Kalnay,  2003).  The  long  term 
average  (climatology)  can  be  viewed  as  a  reference  forecast,  but  like  any  other  forecast, 

it’s  predictability  deteriorates  with  time  (years  to  decades).  In  the  near  term,  weather 
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forecasters  focus  on  specific  atmospheric  phenomena  that  are  generally  anomalies  from 
the  average.  It  is  then  logical  to  use  the  climatology  as  a  baseline  for  predictability. 
Errors  become  “saturated”  (predictability  is  lost)  when  the  error  equals  the  average  error 
of  a  climatology  forecast  (Eckel,  2007). 

2.  Methods  for  Prediction 

The  common  approach  to  atmospheric  prediction  uses  a  deterministic,  single 
solution.  Deterministic  forecasts  have  greatly  improved  with  the  development  of  NWP 
and  since  then,  but  the  inherent  uncertainty  remains  (Kalnay,  2003).  A  consistently 
perfect  deterministic  forecast  is  impossible  to  produce  (Lorenz,  1963).  Eric  Eady  was 
the  first  meteorologist  to  formally  voice  concern  over  the  strict  deterministic  (one 
solution)  method,  in  which  the  future  is  determined  by  the  initial  state  without  an  account 
of  error  in  the  state  (Lewis,  2005). 

The  graphics  product  in  Figure  2  is  a  deterministic  forecast  of  upper  level 
turbulence  (10,000  feet  and  above)  for  the  region  of  interest  applied  in  this  research.  The 
valid  time  (VT)  of  the  graphics  product  is  forecast  hour  24  of  case  two.  The  term 
“Category  II  aircraft”  refers  to  an  instrument  landing  system  capable  of  approaches  to  the 
runway  >  30  meters  above  ground  and  a  Runway  Visual  Range  (RVR)  of  >  360  meters. 
The  product  implies  complete  confidence  in  the  forecast.  Although  it  offers  a 
straightforward  application,  it  neglects  the  importance  and  value  of  forecast  uncertainty. 
Users  may  apply  their  own  subjective  measure  of  uncertainty  to  this  product,  which 
results  in  sub-optimal  decisions  (Wilks,  2006).  Wilks  (2006)  asserts,  “Deterministic 
forecasts  of  future  atmospheric  behavior  will  always  be  uncertain,  and  probabilistic 
methods  will  always  be  needed  to  adequately  describe  that  behavior.” 
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Intensities  of  turbulence  (TURBC):  MDT=moderate,  SVR=severe,  EXTRM=extreme; 
FL=Flight  Level;  CAT  II  ACFT=Category  II  aircraft  [From  USAFE  OWS,  2007], 


An  alternative  is  the  stochastic,  multi-solution  approach,  which  conveys  the 
uncertainty  of  the  forecast  solution  objectively.  “No  weather  forecast  is  complete 
without  a  description  of  this  uncertainty”  (National  Research  Council,  2006).  An 
objective  description  of  the  uncertainty  (potential  forecast  error)  is  necessary  to  avoid 
unnecessary  vulnerabilities,  possible  costly  mistakes,  and  resource  waste.  Quantifying 
uncertainty  can  improve  a  weather  forecast  user’s  ability  to  make  more  economically 
effective  decisions  based  on  their  own  utility  function  (Zhu  et  al.  2002).  Stochastic 
forecasting  does  not  aim  to  correct  the  forecast  error  but  uses  it  to  estimate  error  growth 
and  describe  the  day-to-day,  flow-dependant  forecast  uncertainty  (Eckel,  2007). 


7 


B.  ENSEMBLE  FORECASTING  (EF) 


1.  General  Description 

One  method  to  produce  a  stochastic  forecast  is  to  run  an  ‘ensemble’  of  model 
solutions.  An  ensemble  forecast  (EF)  system  uses  “different  but  concurrent  NWP  model 
runs  with  perturbations  to  the  ICs  and  the  model  to  make  a  spectrum  of  possible 
solutions,  or  a  distribution  of  likely  future  states  of  the  atmosphere”  (Eckel,  2007).  A 
typical  operational  ensemble  model  runs  20  or  more  members  (individual  model  runs), 
each  time  with  a  different  modification  (perturbations  within  the  bounds  of  the  suspect 
error)  of  ICs  and/or  the  model  itself  to  formulate  different  possible  outcomes  (Eckel, 
2007).  Ideally,  each  member  has  an  equal  chance  of  verifying.  The  greater  the  number  of 
members  in  the  ensemble  gives  a  greater  chance  for  the  ensemble  model  to  consistently 
capture  the  truth  (Eckel,  2007).  The  aim  of  EF  is  to  express  the  range  of  possibilities  and 
define  potential  error  in  the  deterministic  forecast  (Eckel,  2007). 

The  graphics  product  displayed  in  Figure  3,  generated  from  the  Joint  Ensemble 
Forecast  System  (JEFS),  is  a  typical  output  from  an  ensemble  model.  JEFS  is  a  multi¬ 
year  pilot  project  directed  by  the  Air  Force  Weather  Agency  (AFWA)  and  the  Fleet 
Numerical  Meteorology  and  Oceanography  Center  (FNMOC)  to  prove  the  value  and 
operational  ability  of  EF  for  enhancing  DoD  operations  and  to  investigate  application  to 
war-fighter  decision  making.  This  prototype  graphics  product  displays  the  probability  of 
turbulence  occurrence  and  is  one  of  the  foundations  of  this  research.  The  upper  left  hand 
image  gives  the  ensemble  consensus  (mean  of  all  the  members)  of  geopotential  heights 
and  winds  speeds.  Using  the  ensemble  data  and  a  turbulence  algorithm,  JEFS  produced 
probabilities  for  light,  moderate,  and  severe  turbulence  thresholds  in  the  following 
images.  Higher  probabilities  are  denoted  by  the  warmer  colors.  Using  decision  science 
(discussed  in  section  C)  an  educated  user  can  objectively  make  decisions  based  upon  the 
probabilities  associated  with  each  turbulence  category. 
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30hr  Forecast 

Valid:  31  JUL  0?,  09Z 

Model  Cycle:  30  JUL  07.  00Z 


Percent  Frob ability  of  Turbulence,  FL340  —  FL390 


Figure  3.  JEFS  Non-Operational  Turbulence  Chart  with  varying  probability  to 
turbulence  occurrence  in  southeast  Asia  [From  JEFS,  2007]. 


2.  Range  of  Possibilities 

Eady  emphasized  consideration  an  ensemble  of  all  possible  atmospheric  solutions, 
and  these  solutions  must  be  expressed  in  terms  of  probabilities  (Lewis,  2005).  The 
atmosphere  can  be  described  as  a  range  of  possibilities  in  a  probability  density  function 
(PDF),  contrary  to  the  singular  deterministic  solution  (Eckel,  2007). 

Like  deterministic  methods,  stochastic  methods  employs  analyses,  forecasts,  and 

climate  data  to  ascertain  the  future  state  of  the  atmosphere,  but  do  so  using  PDFs.  The 

analysis  PDF  is  a  frequency  distribution  of  potential  analyses,  all  of  which  are  random 

samples  from  the  PDF  trying  to  estimate  the  true  PDF  (Eckel,  2007).  The  size  and  shape 

of  the  analysis  PDF  is  a  description  of  the  uncertainty  in  the  analysis  error.  The  forecast 

PDF  works  the  same  way,  but  it  is  a  frequency  distribution  of  potential  forecast  states, 

rather  than  potential  analyses  (Eckel,  2007).  As  a  general  principle,  the  more  narrow  the 
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PDF,  the  more  certain  the  solutions  and  the  more  wide  the  PDF,  the  less  certain  the 
solutions  (Eckel,  2007).  Wider  PDFs  occur  with  the  elapse  of  forecast  lead  time  as  the 
ensemble  members  spread  to  represent  the  growing  error.  As  described  in  section  1, 
analysis  and  model  errors  contribute  to  the  forecast  error  and  inherent  uncertainty. 

Ensembles  produce  forecast  probability  for  a  particular  event  (i.e,  turbulence  > 
MDT)  that  attempt  to  estimate  the  true  probability  from  a  hypothetical  true  PDF  that  is 
relative  to  the  model  system  (Eckel,  2008).  No  one  ‘true  PDF’  exists  for  all  models.  A 
deterministic  forecast  is  a  guess  at  the  unknown,  true  state  of  the  atmosphere  at  a 
particular  moment  found  by  taking  one  sample  from  the  forecast  PDF.  A  perfect 
ensemble  models  all  sources  of  uncertainty  and  takes  infinite  samples  from  the  forecast 
PDF  to  produce  the  “true  PDF”  for  that  particular  model.  Real  world  ensemble  models 
attempt  to  model  many  sources  of  uncertainty  by  taking  limited  samples  from  the  forecast 
PDF.  The  goal  of  ensemble  forecasting  (EF)  is  to  produce  a  forecast  PDF  of  all  possible 
future  states  of  the  atmosphere  from  which  the  true  state  is  consistently  a  random  sample 
(Eckel,  2007). 


3.  Ambiguity 

Ambiguity  is  defined  as  the  uncertainty  of  the  uncertainty  prediction,  which  can 
be  conceptualized  as  errors  bars  about  a  probability  forecast.  Alternatively,  ambiguity 
can  be  considered  as  the  potential  error  in  the  ensemble’s  forecast  PDF  (Eckel,  2008). 
Ambiguity  arises  when  any  contribution  to  forecast  uncertainty  is  not  sufficiently 
accounted  for  in  the  ensemble. 

Several  factors  contribute  to  the  ambiguity  of  ensemble  data,  the  first  of  which  is 
limited  sampling.  Due  to  computational  constraints,  only  so  many  members  can  be 
processed  at  one  time.  Choosing  only  a  few  random  samples  from  a  forecast  PDF  fails  to 
consistently  reflect  the  breadth  of  uncertainty  within  the  true  PDF,  therefore  creating 
ambiguity  (Eckel,  2007).  Another  contribution  to  ambiguity  is  a  coarse  resolution  model. 
The  smallest  atmospheric  motion  wavelength  that  a  model  can  resolve  is  seven  to  eight 
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grid  points  in  length  (Kalnay,  2003).  The  ensemble  cannot  quantify  the  uncertainty  from 
atmospheric  motions  not  represented  in  the  model,  hence  creating  ambiguity  (Eckel, 
2007). 

Additionally,  poorly  simulated  aspects  of  the  IC  and  model  uncertainty  by  the 
ensemble  add  to  the  ambiguity.  A  well-designed  EF  system  attempts  to  models  all 
possible  sources  of  uncertainty  (Eckel,  2007).  The  desire  is  to  have  at  least  different  20 
members  (perturbations)  in  order  to  ensure  the  ensemble  disperses  enough  to  encompass 
the  truth  and  sufficiently  account  for  analysis  and  model  design  uncertainty.  The  term 
‘encompass  the  truth’  describes  the  ability  of  the  ensemble  to  bound  the  verification 
value  with  the  span  of  the  members  (Eckel,  2007).  Three  methods  exist  to  create  model 
perturbations:  1)  IC/boundary  condition  perturbations  2)  ensemble  members  that  have 
different  numerical  schemes  from  the  model  3)  a  combination  of  methods  one  and  two 
(Cunningham,  2006).  Using  only  one  of  first  two  methods  contributes  to  ambiguity.  For 
example,  if  an  ensemble  fails  to  account  for  the  uncertainty  within  the  cumulus  scheme 
by  only  perturbing  IC/boundary  conditions,  then  the  members  will  evolve  cumulus  clouds 
in  the  same  manner  (Eckel,  2007).  The  members  remain  too  similar  to  one  another  as 
time  elapses.  The  insufficient  internal  model  perturbations  lead  to  underdispersion  and  a 
false  indication  of  certainty.  The  unsimulated  variability  of  dispersion  over  time  and 
space  creates  ambiguity. 

Other  possible  contributors  of  ambiguity  include  post-processed  diagnostics  and 
decision  threshold  determination,  all  of  which  will  be  discussed  in  Chapter  III.  The 
challenge  is  to  estimate  this  ambiguity  and  incorporate  it  into  the  decision  making 
process  so  it  does  not  hinder  the  utility  of  the  ensemble  data  (Eckel,  2007). 

As  described  previous  section,  the  ensemble  mean  and  spread  (jue ,  cre )  are 
estimates  of  the  true  PDF  mean  and  spread  (juT,cT).  In  this  research,  the  term  “spread” 

is  used  interchangeably  with  “ensemble  standard  deviation”.  Finite  sampling  hampers 
the  ability  of  an  ideal  ensemble  to  represent  the  true  PDF.  The  random  nature  of  sampling 
causes  the  error  to  be  variable  and  non- systematic,  generating  ambiguity.  The  variability 
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of  / ie  and  cre  about  the  trae  values  (potential  error)  is  described  using  sampling 
distributions  which  are  standardized  with  respect  to  a  normal  PDF  where  /u  =  0  and  cr  =  1 
(Eckel,  2008). 


(1) 


(2) 


Hence,  and  cr,’  are  standardized  potential  errors  in  the  EF  mean  and  variance 

respectively.  Standardizing  the  error  allows  the  error  to  be  proportional  across  the  subsets 
of  data,  making  for  easier  analyzation  and  implementation.  Figure  4  shows  the  sampling 
distributions  from  the  ensemble  mean  (a)  and  spread  (b). 
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(a) 


(Standardized  Error  in  Ensemble  Mean) 


(b) 


Standardized  Ensemble  Standard  Deviation 


(Fractional  Error  in  Ensemble  Spread) 


Figure  4.  Sampling  distributions  of  the  (a)  ensemble  mean  and  (b)  ensemble  spread 
(i.e.,  standard  deviation),  labeled  by  the  number  of  ensemble  members  [From  Eckel, 

2008], 


J.  L.  Devore  determined  the  /je'  distribution  is  Gaussian,  has  no  bias  with  a  standard 
deviation  equaling  <x /  yfn  ,  and  is  independent  of  the  (sampled)  true  PDF  (Eckel,  2008). 

The  ae'  distribution  is  gamma  with  a  spread  dependent  on  shape  of  the  sampled  PDF 

(Eckel,  2008).  The  minimum  error  variance  is  linked  to  finite  sampling  (ensemble  size), 
but  the  error  variance  is  likely  to  increase  due  to  model  deficiencies  (Eckel,  2008). 
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C.  WEATHER  RISK  MANAGEMENT 

1.  Value  and  Cost-Loss  Analysis 

a.  Measure  of  Value 

Weather  forecasts  give  value  to  the  user’s  decision-making  process  when 
the  forecasts  give  the  user  the  ability  to  take  some  action  that  reduces  the  average  long¬ 
term  expense  due  to  an  adverse  weather  event.  Stochastic  forecasts  are  difficult  to  apply 
since  people  often  do  not  think  in  terms  of  probability  (National  Research  Council, 
2006).  Since  most  decisions  to  take  action  are  binary  decisions  (either  “go/no-go”  or 
“protect/don’t  protect),  the  forecast  probability  must  be  converted  to  a  binary  decision. 
Such  a  decision  can  be  arrived  at  by  choosing  a  “probability  decision  threshold”  or  just 
decision  threshold  for  short.  For  example,  an  aggressive  decision  maker  may  use  a 
decision  threshold  of  80%  so  that  he  will  only  decide  the  mission  is  a  no-go  when  the  risk 
of  hazardous  weather  is  greater  than  80%.  A  key  concept  within  this  research  is  that 
there  is  an  optimal  decision  threshold,  which  is  a  particular  to  a  user. 

The  value  score  (V)  helps  illustrates  the  concept  of  the  optimum  decision 
threshold  by  measuring  the  utility  of  the  forecast  in  the  context  of  the  cost-loss  decision 
model  (Wilks,  2006).  To  explain,  each  user  has  a  unique  sensitivity  to  an  event  —  an 
adverse  weather  occurrence  defined  by  a  specific  threshold  (e.g.  50  knots  of  wind  or 
greater).  Cost  (C)  is  the  amount  of  resources  it  takes  to  take  protective  action  for  the 
event,  while  loss  (L)  is  the  amount  of  resources  lost  when  the  event  occurs  without 
protection.  Cost  and  loss  are  not  necessarily  limited  to  monetary  value  but  may  include 
other  resources  such  as  people,  morale,  and  mission  priority.  As  discussed  previously, 
most  decisions  are  binary  so  there  are  four  possible  outcomes  (Table  1).  A  ‘yes’  forecast 
that  verifies  is  hit  (a).  A  ‘yes’  forecast  that  does  not  verify  is  a  false  alarm  (b).  A  ‘no’ 
forecast  that  doe  not  verify  is  a  miss  (c).  A  ‘no’  forecast  that  does  verify  is  a  correct 
rejection  (d).  After  many  forecasts  have  been  complete,  two  quantities  can  be  calculated: 
the  hit  rate  (H)  and  false  alarm  rate  (F),  (Jolliffe,  2003): 
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H  = 


a 


(3) 


a  +  c 


b  +  d 


(4) 


Applying  a  cost-loss  decision  model,  a  decision  maker  can  either  take  protective  action  or 
do  nothing  at  all  with  the  weather  forecast,  incurring  an  associated  expense.  Hence,  from 
these  four  possible  outcomes  come  four  possible  expenses,  which  are  summarized  in 
Table  1. 


ACTION 

TAKEN 

EVENT  OCCURS 

Yes 

No 

C 

C 

Yes 

(a) 

(b) 

No 

L 

Zero  Expense 

(c) 

(d) 

Table  1.  Expense  Matrix:  Costs  (C)  and  Losses  (L)  for  different  outcomes  in  C/L 
decision  model  [From  Table  8.1,  Jolliffe,  2003], 


The  value  score  (V)  of  the  forecast  is  a  function  of  the  quality  of  the 
forecast  system  (H  and  F),  the  user’s  C/L  (a),  and  the  climatology  rate  (s).  The 
climatology  rate  is  simply  the  average  rate  of  occurrence  of  the  weather  event  derived 
from  past  records  of  the  event,  independent  of  forecaster  input.  Jolliffe  (2003)  gives  the 
value  score  equation: 

y  _  min(a,s)-F(l-s)a+Ffa(l-a)-,s'  (5) 

min(a,,s)-,sa 
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A  value  of  V  =  1  signifies  a  perfect  forecast,  while  V  <  0  signifies  that  climatology 
forecasts  give  better  decision  input.  The  highest  possible  V  for  a  particular  user  is 
achieved  when  the  decision  threshold  is  equal  to  the  C/L  (Figure  5),  thus  indicating  the 
optimal  decision  threshold. 


Figure  5.  Variation  of  value  V  with  decision  threshold  (pt)  for  ECMWF  EPS 

probability  forecasts  for  users  with  C/L  =  0.2  (solid  line)  and  C/L  =  0.8  (dashed  line) 

[From  Jolliffe,  2003], 


To  demonstrate  this  concept,  consider  the  following  example.  Two 
different  users  have  the  same  expenses  for  the  event  of  surface  winds  greater  than  50 
knots:  C  =  $150K,  L  =  $1,000K;  hence  the  user’s  C/L=0.15.  User  A  is  hesitant  and 
takes  action  to  protect  anytime  the  FP  is  greater  than  0%.  User  B  is  daring  and  will  only 
take  action  when  the  FP  is  100%.  Both  users  are  assumed  to  know  nothing  about 
decision  science.  The  forecasting  system  is  perfectly  reliable,  so  the  number  of  event 
occurrences  on  average  (observed  rate  of  the  event)  is  equal  to  the  product  of  the  forecast 
probability  (FP)  and  number  of  forecast  cases.  The  following  expenses  (E)  are  calculated 
for  three  different  sets  of  100  forecast  cases  where  >  50  knots  of  wind  was  possible. 
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FP  for  >  50  knots  =  .20 


User  A  (alwavs  protect): 

E  =  (100  cases)  *  ($150Kcost) 

=  $15,000K 

WINNER 

User  B  (never  protect): 

E  =  0.20  *(100  cases)  *  ($1,000K  loss) 

=  $20,000K 

FP  for  >  50  knots  =  .10 

User  A  (alwavs  protect): 

User  B  (never  protect): 

E  =  (100  cases)  *  ($15  OK  cost) 

E  =  0.10  *(100  cases)  *  ($1,000K  loss) 

=  $15,000K 

=  $10,000K 

WINNER 

FP  for  >  50  knots  =  .15 

User  A  (always  protect):  User  B  (never  protect): 

E  =  (100  cases)  *  ($150K  cost)  E  =  0.15  *(100  cases)  *  ($1,000K  loss) 

=  $15,000K  =  $15,000K 


«-TIE* 


In  this  last  scenario,  the  total  expense  is  equal  for  both  users  when  the  FP 
=  C/L  =  0.15.  User  B  expense  is  less  than  user  A  expense  only  when  FP  <  C/L. 
Therefore,  an  astute  user  would  act  like  User  B  (take  no  protective  action)  when  FP  < 
C/L  and  would  act  like  User  A  (take  protective  action)  when  FP  >  C/L.  Over  the  long 
term,  a  user  can  minimize  expense  and  yield  the  greatest  value  by  noting  that  a  decision 
threshold  =  C/L  is  the  optimal  decision  threshold. 

Realistically,  some  users  may  find  it  difficult  to  define  their  C/L  when 
their  expenses  are  too  complex  or  difficult  to  quantify.  A  user  can  still  attain  value  from 
the  forecast  by  using  a  decision  threshold  within  the  “ballpark”  of  their  actual  C/L  ratio. 
As  seen  in  Figure  5,  a  decision  threshold  approximately  between  0.02  and  0.45  still  yields 
value  for  the  C/L=0.2  user.  This  concept  is  important  to  the  integration  of  stochastic 
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forecasting  into  DoD  operations.  Users  can  still  attain  value  from  an  ensemble  forecast 
without  having  to  invest  resources  for  an  in-depth  analysis  to  determine  their  actual  C/L 
ratio. 

It  is  important  to  note  that  no  single  decision  threshold  is  optimal  for  a 
range  of  users  with  different  measures  of  risk.  For  example,  if  the  C/L=0.8  user,  who 
has  a  high  tolerance  to  risk,  acts  on  C/L=0.2  user’s  optimal  decision  threshold,  he  will 
attain  negative  value  from  the  forecast  and  miss  opportunities  to  act  by  protecting 
unnecessarily.  The  optimum  decision  threshold  =  C/L  concept  only  applies  to  properly 
calibrated  forecast  probabilities;  the  user  cannot  simply  optimize  his/her  decision 
threshold  otherwise. 

When  the  FP  exceeds  the  decision  threshold,  protective  action  should  be 
taken.  In  this  way,  the  probability  forecast  is  converted  to  a  binary  decision  input 
(Jolliffe,  2003). 

b.  Risk  Tolerance  and  Cost-Loss  Ratio 

A  cost- loss  analysis  (C/L  ratio)  is  a  way  to  quantify  the  user’s  tolerance  to 
Risk,  or  ‘risk  tolerance’.  The  risk  tolerance  is  a  measure  of  the  amount  the  risk  the  user  is 
willing  to  accept.  For  example,  a  user  with  a  small  C/L  has  relatively  large  potential 
losses  and  benefits  by  taking  action  at  relatively  low  forecast  probabilities.  Users  with 
small  C/L  are  highly  sensitive  to  adverse  weather  events;  hence,  their  risk  tolerance  (RT) 
is  lower.  Conversely,  a  user  with  a  high  C/L  needs  a  higher  probability  to  justify  taking 
protective  action  for  an  event  since  the  cost  of  protection  is  not  much  less  than  the 
potential  loss.  Users  with  high  C/L  are  more  tolerant  to  adverse  weather  events;  hence, 
their  RT  is  higher.  In  this  research,  the  term  ‘RT  threshold’  is  interchangeable  with  the 
term  ‘decision  threshold’.  The  term  ‘mission  threshold’  describes  those  weather  event 
criteria  that  restrict  mission  accomplishment  and  may  pose  a  threat  to  safety.  The  best 
approach  is  to  combine  knowledge  of  mission  thresholds  and  RT  with  a  skilled  stochastic 
forecast  to  give  an  optimum  decision  input  that  minimizes  loss  and/or  maximizes  gains 
(Eckel,  2007). 
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2.  Operational  Risk  Management 


In  addition  to  economic  benefit,  military  decision  makers  are  also  concerned  with 
risks  such  as  loss  of  life  and  benefits  such  as  mission  success  (Cunningham,  2006).  The 
ability  of  stochastic  forecasting  to  quantify  uncertainty  serves  to  strengthen  the 
operational  Risk  Management  (ORM)  system.  The  foundation  of  ORM  is  to  accept  risk 
when  the  potential  benefits  are  greater  than  the  cost  and  avoid  risk  when  potential  losses 
are  unaffordable  in  order  to  maximize  gain/minimize  loss  in  the  long  run  (AFI  90-901; 
Zhu  et  al.  2002).  Two  types  of  ORM  exist:  offensive  ORM  acts  to  maximize  gains 
(benefits)  and  defensive  ORM  acts  to  minimize  loss  (risk).  From  the  cost-loss  analysis 
perspective,  the  C  and  L  are  defined  in  terms  of  benefits  and  risks  respectively.  In  order 
for  the  concept  of  stochastic  forecasting  to  be  accepted  and  implemented  by  the  DoD, 
then  it  has  to  be  part  of  the  ORM  decision-making  process  (Eckel  et  al,  2008). 

The  Air  Force  (AF)  defined  risk  management  guidelines  in  AF  Instruction  90-901 
and  elaborated  the  guideline  in  AF  Pamphlet  90-902.  “The  USAF  aim  is  to  increase 
mission  success  while  reducing  the  risk  to  personnel  and  resources  to  the  lowest  practical 
level  in  both  on-  and  off-duty  environments”  (AFP AM  90-902).  ORM  has  become  a 
mode  of  thinking  in  the  AF  with  yearly  training,  safety  days,  and  in  everyday  mission 
checklists  (Cunningham,  2006).  The  entire  set  of  AF  risk  management  goals  are  listed  in 
Figure  6.  Along  with  these  goals,  are  also  4  guidelines  1)  accept  no  unnecessary  risk,  2) 
make  decisions  at  the  appropriate  level,  3)  accept  risk  when  benefits  outweigh  the  cost,  4) 
integrate  ORM  into  Air  Force  doctrine  at  all  levels  (AFI  90-901).  Other  DoD  service 
components  follow  similar  ORM  principles. 
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Figure  6.  U.S.  Air  Force  Risk  Management  Goals  [From  AFP  AM  90-902], 

The  ability  of  a  stochastic  forecast  to  quantify  uncertainty  is  key  to  giving  the 
DoD  decision-maker  a  complete  understanding  of  the  weather  situation.  Deterministic 
forecasts  omit  uncertainty,  while  stochastic  forecasting  objectively  measures  uncertainty. 
This  objective  measure  allows  the  user  to  apply  their  own  set  of  risk  tolerances  to  make 
an  optimized,  unbiased  decision  to  minimize  risk  and/or  maximize  benefits.  Ensemble- 
based  probabilistic  forecasts  have  a  place  in  DoD  operations  by  incorporating  vital 
information  into  the  decision-making  process. 

D.  CALCULATING  AND  COMMUNICATING  FORECAST  TURBULENCE 

AND  ITS  UNCERTAINTY 

1.  Today’s  Deterministic  Methods 

High  altitude  clear-air  turbulence  (CAT)  can  be  an  extremely  detrimental  weather 
event  for  aviation.  The  number  of  fatal  accidents  involving  CAT  encounters  are 
relatively  low  (McLean,  1986),  but  serious  injury  and  structural  damage  have  occurred 
with  severe  (SVR)  to  extreme  CAT.  Rerouting  and  loitering  due  to  CAT  and  flying 
through  CAT  increases  fuel  expense  and  mission  times  (Ellrod  and  Knapp,  1992). 


20 


Significant  CAT  is  prevalent  in  areas  of  vertical  wind  shear,  horizontal  shear, 
convergence,  horizontal  deformation,  lapse  rate  discontinuities,  and  strong  horizontal 
thermal  gradients  (Ellrod  and  Knapp,  1992).  NWP  models  cannot  practically  predict 
turbulence  because  the  resolution  required  for  aircraft  scale  turbulence  remains  too  high. 
Diagnostics  and  rules  of  thumb  (ROT)  have  been  developed  to  produce  turbulence 
forecasts  by  correlating  synoptic  and  mesoscale  patterns  to  CAT  occurrences 
(Cunningham,  2006).  The  current  mainstream  practice  of  forecasting  turbulence  is 
largely  deterministic.  Operational  forecasters  use  a  combination  of  tools  like  raw  model 
data,  model-derived  turbulence  diagnostics,  and  forecast  ROT  with  their  skill  and 
experience  to  produce  turbulence  forecasts  like  the  one  illustrated  in  Figure  2,  which  is  an 
example  of  the  most  basic  decision  input  in  which  objective  uncertainty  is  ignored. 

The  Integrated  Weather  Effects  Decision  Aid  (IWEDA)  is  a  baseline  decision 
input  tool  that  compares  mission  limiting  weather  criteria  with  deterministic  forecasts  to 
produce  decision  inputs.  IWEDA  has  marginal  and  unfavorable  thresholds  established 
for  different  assets  (resources  such  as  people  or  equipment)  depending  on  the  assets’ 
vulnerability  to  the  weather  parameter.  IWEDA  defines  the  mission  threshold  as  “the 
point  where  the  occurrence  of  a  meteorological  element  causes  a  significant  (moderate  or 
severe)  impact  on  a  military  operation,  system,  subsystem,  or  personnel”  (Shirkey  and 
Gouveia,  2002).  IWEDA  determined  the  mission  thresholds  by  “doctrine,  safety,  or 
engineering  factors  (people,  modeling,  or  testing)”  (Shirkey  and  Gouveia,  2002). 

Decision  inputs  are  color  coded.  Red  indicates  conditions  in  which  operations 
“are  severely  impacted:  There  is  either  a  total  or  severe  degradation  or  the  operational 
limits  or  safety  criteria  have  been  exceeded”  (Shirkey  and  Gouveia,  2002).  Amber 
indicate  conditions  in  which  operations  “are  marginal  and  the  operational  capability  is 
degraded,  or  there  is  a  marginal  degradation”  (Shirkey  and  Gouveia,  2002).  Personnel 
can  still  proceed  with  operations,  but  need  to  proceed  with  caution.  Green  indicates  that 
no  operational  restrictions  exist  (Shirkey  and  Gouveia,  2002).  To  give  an  example  of  an 
unfavorable  (red)  mission  threshold:  personnel  cannot  safely  perform  a  static  line 
parachute  jump  in  greater  than  13  knots  of  surface  wind.  The  marginal  (amber)  threshold 
is  9  knots. 
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IWEDA  in  a  sense  is  giving  a  hint  of  uncertainty  information  with  this  marginal 
(amber)  mission  thresholds  by  inferring  an  average  forecast  error  of  4  knots.  IWEDA 
includes  no  objective  uncertainty  information.  Other  limitations  of  IWEDA  include  no 
consideration  of  the  user’s  risk  tolerance  and  complete  ignores  ambiguity. 

2.  WRAP 

Methods  to  incorporate  objective  uncertainty  into  the  decision  making  process  are 
in  development.  One  approach  to  produce  and  visualize  stochastic  weather-based 
decision  input  is  the  prototype  Weather  Risk  Analysis  and  Portrayal  (WRAP)  tool.  The 
Army  Research  Laboratory  funded  Next  Century  Corporation  to  develop  the  WRAP  tool 
to  address  the  need  of  military  users  to  correctly  and  efficiently  interpret  ensemble 
forecasts  (Next  Century,  2007).  WRAP  processes  ensemble  data,  users’  RT,  and 
IWEDA’s  mission  thresholds  to  create  a  decision  input  which  is  superior  to  IWEDA. 
WRAP  uses  objective  uncertainty  information  to  formulate  unique  decision  inputs  that 
vary  with  different  risk  tolerances  whereas  IWEDA  uses  subjective  uncertainty 
information  to  generate  generic  decision  inputs.  WRAP  defines  the  user’s  RT  as  the 
level  of  risk  the  user  is  willing  to  accept  (Next  Century,  2007).  WRAP  uses  the  familiar 
stop-light  (red,  amber,  green)  format  in  their  graphics  product  to  easily  convey  the 
user-specific  decision  input.  A  user  with  a  high  risk  tolerance  is  going  to  have  different 
decision  input  that  a  user  with  a  low  risk  tolerance  (Figure  7). 
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Figure  7.  Examples  of  decision  input  visualizations  from  (a)  medium  risk  tolerant 
user  and  (b)  from  low  risk  tolerant  user.  Adapted  from  the  Weather  Risk  Analysis  and 
Portrayal  Phase  II  Prototype  Risk  Tolerance  Chart.  [From  Next  Century,  2007], 


The  example  below  illustrates  WRAPs  methodology  for  determining  the  red  (no- 
go),  amber  (marginal),  or  green  (go)  colored  decision  inputs.  From  the  previous  IWEDA 
example,  the  weather  parameter  of  concern  is  surface  wind  for  a  static  line  parachute 
jump  into  a  drop  zone.  The  user  has  a  medium  tolerance  to  risk  with  a  RT  of  40%.  The 
PDFs  displayed  in  Figure  8  represent  the  range  of  possible  outcomes  for  surface  wind 
speed  as  defined  by  an  ensemble  forecast.  The  risk  of  exceeding  the  marginal  or 
unfavorable  thresholds  is  the  area  under  the  PDF  to  the  right  of  the  threshold. 

In  Figure  8a,  nearly  all  of  the  PDF  lies  below  the  marginal  threshold.  The  risk  of 
>  marginal  and  >  unfavorable  thresholds  is  one  and  zero  percent  respectively.  Since  the 
RT  exceeds  the  risk  for  either  threshold,  the  user  is  given  a  “green”  decision  input.  In 
Figure  8b,  the  majority  of  the  PDF  lies  above  the  marginal  threshold  (99%).  Since  the 
risk  of  >  unfavorable  threshold  (the  worse  condition)  is  68%  and  exceeds  the  RT,  the  user 
is  given  a  “red”  decision  input.  In  Figure  8c,  the  majority  of  the  PDF  lies  above 
marginal  threshold.  The  risk  of  >  unfavorable  conditions  is  1 1  percent.  The  risk  of  > 
marginal  threshold  is  63  percent.  Since  this  risk  exceeds  the  RT,  the  user  is  given  a 
“amber”  decision  input. 
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a) 


Marginal  Threshold:  9kt  Unfavorable  Threshold:  131ct 


Decision  Input 


b) 


c) 


Marginal  Threshold*  9kt  Unfavorable  Threshold:  13kt  Decision  Ilipilt 

—  0 


Figure  8.  PDF  representation  of  the  risk  of  exceeding  marginal  and  unfavorable 
thresholds  using  a  RT  =  40%.  a)  Green  decision  input,  b)  Red  decision  input,  c)  Amber 

decision  input.  [After  Eckel,  2006]. 

WRAP  is  limited  by  its  use  of  the  deterministic  IWEDA  marginal  thresholds  in  a 

stochastic  context  to  derive  ambiguity.  Using  the  example  from  the  static  line  parachute 

jump,  WRAP  is  giving  a  hint  of  ambiguity  information  using  two  thresholds.  The 

marginal  threshold  (>  9)  infers  there  is  an  average  forecast  error  of  four  knots  for  the 

unfavorable  mission  threshold  (>13  knots).  WRAP  makes  no  formal  calculations  to 

determine  the  ambiguity,  rather  it  uses  the  marginal  threshold  as  a  proxy.  The  concept  of 
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marginal  threshold  only  makes  sense  in  a  deterministic  context.  The  amber  regions  on 
stochastic  weather-based  decision  input  graphic  should  denote  regions  of  “too  close  to 
call”  not  regions  of  “proceed  with  caution.”  Only  one  threshold,  the  mission 
(unfavorable)  threshold,  should  be  applied  to  the  stochastic  context. 

3.  Conclusion 

Today’s  method  to  give  decision  input  is  deterministic  with  no  uncertainty 
information  or  consideration  of  RT  included.  IWEDA  is  a  more  advanced  deterministic 
tool  with  its  application  of  marginal  and  mission  thresholds  to  give  decision  inputs. 
WRAP,  the  prototype  stochastic  weather  decision  aid  improves  upon  IWEDA,  by 
incorporating  both  forecast  probability  uncertainty  and  RT  but  poorly  estimates 
ambiguity  and  does  not  consider  the  uncertainty  in  RT.  This  research  demonstrates  the 
same  capabilities  of  WRAP,  but  objectively  estimates  ambiguity  and  subjectively 
estimates  RT  uncertainty  to  incorporate  the  possibly  vital  information  into  the  decision 
making  process. 
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III.  METHODOLOGY 


Using  the  previous  background  research  as  a  foundation,  this  study  demonstrates 
capabilities  that  will  serve  to  further  education,  development,  and  utilization  of  stochastic 
methods  in  the  Air  Force  and  Department  of  Defense.  Generating  products 
communicating  flight  level  turbulence  risk  to  the  user  for  optimal  decision  making 
requires  a  three  part  process,  outlined  in  Figure  9.  Specifically,  part  one  generated  non- 
calibrated  forecast  probabilities  of  varying  levels  of  turbulence  intensities  using  the 
uniform  ranks  method.  Part  two  generated  calibrated  forecast  probability,  ambiguity 
(using  the  Taijitu  method),  and  decision  input  data  by  comparing  the  risk  and  the  risk 
tolerance.  Part  three  introduced  a  user-friendly  graphical  user  interface  (GUI)  to  visualize 
the  information  from  Part  two. 

The  researcher  used  Matlab,  version  7.0  or  later,  as  the  computer  programming 
platform.  The  study’s  methods  can  be  applied  to  other  computer  programming  platforms 
as  well,  especially  in  the  visualization  realm.  Matlab  has  limited  graphics  capability.  A 
more  sophisticated  programming  platform  can  produce  a  more  advanced  visualization  to 
communicate  flight  level  uncertainty. 
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Download,  de-Grib  Raw  GFS  Global  Ensemble  Data,  and  save  flat  files 


'Part  I:  Generate  forecast  probability  (FP)  for  varying  levels  of  turbulence  intensity 

1)  Construct  L at/Lon  grid,  reorient  GFS  ensemble  data  grid 
For  each  member  of  one  forecast  hour  (FH): 

2)  Compute  Ellrod/Knapp  Turbulence  Index  (TI)  over  global  grid 
Interpolate  from  millibar  to  flight  levels  (FL) 

Cut  grid  to  EUCOM  AOR 

Combine  all  members'  TI  and  calculate  FP  for  LGT,  MDT,  S VR  turbulence 
Validate 


3) 

4) 

5) 
,6) 


^Part  II:  Generate  Decision  Input  (PI),  calibrated  FP,  max  potential  error  (PE)  output 

1)  Acquire  500mb  verification  data  for  GFS  ensemble  and  make  error  variance  calculations 

2)  Formulate  ambiguity  intervals  (error  bars)  using  Taijitu  method 

3)  Formulate  risk  tolerance  (RT)  error  bars 

4)  Process  FP  data  from  Part  1  through  decision  input  (DI)  algorithm  to: 

a)  Add  calibration  factor  to  FP 

b)  Establish  error  bars  to  calibrated  FP  using  step  2  data. 

c)  Compare  FP  and  RT  error  bars  to  assign  either  risk  unacceptable/“no-go”  (red),  inconclusive 
(amber),  or  risk  acceptable/“go”  (green)  decision  input 

,  d)  Compute  maximum  potential  error  (PE) 


Part  III:  Generate  Graphical  User  Interface  (GUI)  to  display  visualizations  of  DI,  calibrated  FP,  and 

maximum  PE  using  interactive  inputs  of  RT,  mission  limiting  turbulence  intensity  (INT),  FL,  and 
desired  FH  from  user 


Figure  9.  Three  part  process  of  generating  products  communicating  flight  level 

turbulence  risk 


A.  PART  ONE 

1.  Model  Description  and  Ingest 

The  Global  Ensemble  Forecasting  System  (GEFS),  used  in  this  study,  is  a 

worldwide  (360  x  181)  model  grid  generated  by  the  National  Centers  for  Environmental 

Prediction  (NCEP).  The  GEFS  dataset  has  21  members,  one  degree  grid  spacing,  50 

millibar  (mb)  pressure  increments,  six  hour  forecast  intervals,  and  a  384  hour  forecast 

length.  In  this  research,  the  maximum  forecast  length  is  84  hours.  The  50  mb  pressure 

increment  essential  for  calculating  the  turbulence  index  is  only  available  up  to  forecast 
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hour  (FH)  84.  The  GEFS  uses  an  ensemble  transform  to  create  initial  condition 
perturbations;  no  internal  model  perturbations  are  applied. 

GEFS  is  available  in  gridded  binary  version  two  (GRIB2)  format  on  the 
GEFS  ftp  website.  For  the  purposes  of  the  study,  three  variables  (3  GRIB2  files)  for  each 
member  were  downloaded  and  “de-gribbed”  into  a  usable  format:  u  and  v  components  of 
the  wind  (meters/second),  and  the  geopotential  heights  (meters).  The  latitude,  longitude, 
and  variable  spatial  grids  were  reoriented  so  the  coordinate  90°N  and  0°E  is  positioned  at 
the  upper  left  hand  comer  of  the  grid.  In  order  to  explore  risk  analysis  for  different 
amounts  of  turbulent  areas,  the  researcher  downloaded  two  GEFS  model  runs:  one  taken 
from  the  fall  season  (02  OCT  07,  00Z)  and  one  taken  from  the  winter  season  (30  JAN  08, 
00Z).  A  single  sample  forecast  hour,  flight  level,  forecast  probability,  and  risk  tolerance 
from  the  winter  case  will  be  used  throughout  this  chapter  to  exhibit  the  methodology. 

2.  Turbulence  Index  (TI)  Calculation 

The  Ellrod  and  Knapp  (1992)  turbulence  index  (TI)  calculation  is  the  diagnostic 
of  choice  in  this  study  because  of  its  skill  and  operational  history  with  NCEP’s  and 
AFWA’s  model  derived  turbulence  forecasts.  The  Ellrod  TI  was  developed  from 
concepts  derived  from  30  years  of  turbulence  studies  starting  in  the  1950s  (Ellrod  and 
Knapp,  1992).  The  physical  foundation  uses  the  concept  of  frontogenic  intensity. 
Frontogenesis  increases  vertical  wind  shear  (VWS)  and  therefore  increases  the  threat  of 
clear  air  turbulence  (CAT)  (Ellrod  and  Knapp,  1992).  The  product  of  VWS  with  the 
combination  of  horizontal  deformation  (DEF)  and  convergence  (CVG)  forms  the  basis 
the  Ellrod  TI  equation  (Ellrod  and  Knapp,  1992): 


TI  =  VWS  *  (DEF  +  CVG )  (6) 

The  VWS  term  describes  a  rapid  change  in  wind  speed  and/or  direction  with 
height  (Ellrod  and  Knapp,  1992).  The  equation  for  VWS  is: 


VWS  = 


(At/2  +Av2)1/2 
A z2 


(7) 
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where  A u  and  Av  are  the  differences  of  wind  speeds  between  grid  points,  and  A z  is  the 
difference  in  geopotential  heights  (layer  thickness). 

The  DEF  term  is  characterized  by  a  “property  of  a  fluid  that  transforms  a  circular¬ 
shaped  area  of  fluid  to  an  elliptical  shape”  (Ellrod  and  Knapp,  1992).  The  equation  for 
DEF  is: 

DEF  =  (DST 2  +  DSH  2J'2  (8) 

where  DST  is  the  stretching  deformation  and  DSH  is  the  shearing  deformation,  given  by 
the  following  equations: 


dx 


dv 

dy 


(9) 


DSH=^  +  ^ 

dx  dy 


(10) 


Ellrod  and  Knapp  (1992)  describe  CVG  as  “a  compaction  of  a  fluid  caused  by 
the  confluence  of  streamlines  and/or  deceleration  of  air  parcels.”  CVG  is  the  negative  of 
divergence  and  is  given  by  the  following  equation: 


f 

CVG  =  - 


du  dv'' 

- 1 - 

dx  dyy 


(11) 


AFWA  uses  the  Ellrod  index  operationally  in  an  algorithm  to  automatically  compute 
turbulence  forecasts  from  the  Mesoscale  Model  (MM5)  (G.  Brooks,  2008,  personal 
communication). 

The  researcher  acquired  and  manipulated  AFWA’s  turbulence  algorithm  to 
construct  the  algorithm  used  in  this  research.  The  researcher  used  a  three  pressure  level 
center  differencing  approach  where  Az  is  computed  at  one  pressure  level  (p2)  using  the 
data  from  the  pressure  level  above  (p3)  and  below  (pi)  and  divided  by  the  total  depth 
(Figure  10).  The  TI  for  the  top  pressure  level  was  computed  using  one  side  differencing. 


30 


The  TI  for  the  bottom  pressure  level  (850  mb)  was  negated  because  it  was  of  little 
consequence  to  the  interpolation  scheme,  discussed  in  the  next  section. 


♦ 


& - - P3 


Az - 0> - -0- - 0- - p2 


e 


Pi 


< 


Ax 


Figure  10.  Three  pressure  level  computation  method.  Red  line  indicates  location  of 
pressure  level  computation.  Circled  points  of  data  used  in  calculation  of  TI  at  square 
point.  Note  Ay  is  not  shown,  but  perpendicular  to  the  page. 

The  horizontal  distance  between  the  grid  points  was  computed  for  the  one 
degree  global  grid  using  the  equations: 


(2*n 


8x  =  - -  *cos(degree  of  latitude) 

V  360  ) 


(12) 


(13) 


where  Re  is  the  average  earth  radius  =  6,372,797  meters.  The  end  TI  result  was 
multiplied  by  the  scaling  factor  lxlO7  (Ellrod  and  Knapp,  1992). 

The  Ellrod  index  thresholds  for  turbulence  intensity  are  dependant  on  the  model 
used.  The  study’s  TI  thresholds  were  taken  from  the  JEFS  Joint  Global  Ensemble  (JGE) 
upper  level  turbulence  algorithm  which  is  also  applied  to  the  GFS  ensemble.  The 
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intensity  thresholds  for  light  (LGT),  moderate  (MDT),  and  SVR  (SVR)  turbulence  are  2, 
5,  10  (unitless)  respectively  (G.  Brooks,  2008,  personal  communication). 

3.  Interpolation 

After  the  TI  was  calculated,  the  researcher  interpolated  from  pressure  level  data  to 
constant  flight  levels.  Interpolation  is  a  necessary  step  to  convert  the  turbulence  data  to 
standard  flight  levels  so  the  data  can  easily  be  applied  to  DoD  operations.  The 
interpolated  heights  were  chosen  with  consideration  of  the  logarithmic  pressure  spacing 
of  the  atmosphere.  Since  the  GEFS  data  levels  are  incremented  every  50  millibars,  the 
researcher  chose  a  logical  height  level  scheme  to  minimize  over  interpolation  and  still 
yield  useful  flight  level  products.  The  flight  levels  above  20  thousand  feet  are 
incremented  every  five  thousand  feet  and  flight  levels  between  10  and  20  thousand  feet 
are  incremented  every  two  thousand  feet  (MSL)  (Figure  1 1). 

Matlab  has  routines  to  automatically  interpolate  data  from  one  height  to  another. 
The  Matlab  one-dimensional  interpolation  routine  (interpl)  has  several  options  to  include 
‘linear’  and  ‘spline.’  The  linear  method,  as  it  states,  interpolates  data  linearly  from  the 
original  level  to  the  desired  level.  The  spline  method  interpolates  data  using  a  cubic 
spline  (Figure  12),  which  is  constructed  using  third  order  piecewise  polynomials 
(Weisstein,  2008). 

To  compare  the  performance  of  the  methods,  the  researcher  took  a  group  of  cross 
sections  lying  within  a  strong  TI  gradient  in  WRN  CONUS  from  the  same  model  day, 
time,  and  forecast  hour  pictured  in  Figure  14.  The  two  methods  were  interpolated  to  the 
research  height  levels  (Figure  13).  Interpolated  TI  results  for  both  methods  were  similar 
below  25,000  feet.  Larger  differences  occur  above  25,000  feet  because  of  larger  distances 
exist  between  the  original  data  points  and  the  interpolate  points  due  to  logarithmic 
spacing.  Since  the  upper  level  winds  generally  do  not  decrease  and  increase  with  height 
linearly,  the  spline  method  interpolated  the  TI  in  between  the  large  gaps  (separating  the 
original  data  points)  more  realistically.  For  this  reason,  the  researcher  implemented  the 
spline  method. 
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50000 


Figure  1 1 .  Skew-T  Log-P  diagram  showing  50  mb  increment  pressure  level  data 
locations  (red  text  and  lines)  and  interpolated  levels  (blue  text  and  lines).  For  reference, 
standard  atmospheric  heights  are  displayed  (green  text). 


Figure  12.  Piecewise  cubic  spline  (black)  and  piecewise  linear  segments  (red)  [From 

Weisstein,  2008] 
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After  the  GEFS  model  ingest,  TI  calculation,  and  interpolation  of  height  levels, 
the  resultant  output  was  TI  (no  units)  data  for  all  21  members.  Member  number  two  from 
the  January  31,  2008  00Z  run  is  shown  in  Figure  14. 

Because  of  the  extremely  large  amount  of  data,  the  researcher  cut  the  global  grid 
(181°  x  360°)  down  to  a  specific  area  of  responsibility  (AOR),  the  European  Command 
(EUCOM),  which  measures  (41°  x  71°).  The  EUCOM  AOR  serves  a  primary  location  for 
refueling  and  transport  missions  supporting  Middle  East  Operations,  which  are  particular 
vulnerable  to  upper  level  turbulence. 
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a) 


-5  0  5  10  15  20  25  30  35  40  45 

Tl  (unitless) 


Figure  13.  Interpolated  TI  data  using  “linear”  and  “spline”  methods  versus  the 
original  TI  data,  Latitude/Longitude:  a)  35°N/245°W  b)  36°N/243°W  c)  33°N/239°W. 
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2008013000  FH:  024,  Member:  02  Turbulence  Index  (no  units),  FL:  350  (MSL) 


Figure  14.  Example  of  raw  TI  output  on  a  global  grid.  FH:  024  =  Forecast  Hour  24  , 

FL350  (MSL)  =  Flight  Level  35,000  feet  (Mean  Sea  Level) 


4.  Forecast  Probability  Calculations 

To  calculate  forecast  probability  (FP),  the  most  straightforward  method  is  the 
counting  method,  which  biases  FP  toward  the  extreme  probabilities  (Eckel,  2007).  To 
convert  the  2 1  member  TI  output  to  forecast  probability,  the  researcher  employed  a  more 
advance  method  called  uniform  ranks.  The  ensemble  members’  output  of  TI  at  one  grid 
point  is  assumed  to  be  equally  likely  and  ranked  ordered  (sorted  from  least  to  greatest). 
Uniform  rank’s  total  probability  is  divided  into  n+ 1  bins  (n=total  number  of  members) 
which  equal  the  number  of  possible  ranked  positions  of  the  threshold  when  combined 
with  the  members  (Hamill  and  Colucci,  1997).  The  rank-probability  bins  that  exceed  the 
threshold  value  are  summed  to  produce  the  probability  of  the  turbulence  exceeding  the 
threshold. 
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The  uniform  ranks  method  considers  the  probability  existing  partially  in  adjacent 
bins  by  linearly  interpolating  the  distance  between  the  threshold  value  and  value  in  the 
adjacent  bin.  Eckel  (1998;  2003)  gives  the  probability  of  the  partial  bin: 


P(T<V<X')  =  ( 


■V,  ~  T 

Xi+-  -X; 


m,-.  ■ 


(14) 


In  the  preceding  equation,  T  is  the  threshold  value,  V  is  the  verification  value,  xi  is  the 
value  of  the  ensemble  member  with  rank  i,  xi+i  is  the  value  of  the  ensemble  member  of 
i+1,  and  RPi+i  is  the  amount  of  the  probability  of  the  verification  rank  i+1  (that  is 
RPi+i=l/(n+l)).  Using  the  uniform  ranks  method,  the  following  equation  calculates 
forecast  probability.  Eckel  (1998,  2003)  adapted  the  uniform  ranks  method  from  Hamill 
and  Colucci  (1997). 

FP  -  P(T  <V  <  x. )  4-  {ftoftnembers  >  T )  /(«  - 1))  (15) 

In  the  following  example  (Figure  15),  a  ten-member  ensemble  is  used.  The  output 
parameter  is  a  turbulence  index  (TI).  The  threshold  is  moderate  (MDT)  turbulence, 
measured  by  TI  equal  to  or  greater  than  five.  The  resultant  forecast  probability  is  77%. 


Figure  15.  Uniform  Ranks  Method:  calculating  turbulence  FP  >  MDT  (TI>5)  using 

10-member  ensemble. 

When  T  is  ranked  n+ 1,  the  fraction  of  probability  is  computed  by  assuming  the 
total  probability  falls  into  the  upper  (right)  tail  of  a  Gumbel  distribution  (Eckel  1998, 
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2003).  The  Gumbel  cumulative  density  function  (CDF)  gives  the  ability  to  characterize 
extreme  probability  events  (Wilks  2006),  which  is  given  in  the  following  equation: 


F(x)  =  exp:' 


exp-: 


(*-£) 


p 


(16) 


The  Gumbel  distribution  parameters  are  estimated  using  the  statistical  method  of 
moments,  which  employs  the  sample  mean  and  standard  deviation  (Wilks,  2006): 


P  = 


K 


£  =  x-  }f 


(17) 


(18) 


y  =  0. 5 772 1 ,  (Euler’s  Constant) 


(19) 


With  extreme  low  probability  situations  (where  T  falls  in  the  highest  bin),  the  forecast 
probability  is  calculated  using  the  following  equation  adapted  from  Eckel  (1998,  2003): 


FP  =  P(T  <V)  = 


f  1  -  F(T)  ' 

1  ~FM. 


RP, 


n+ 1 


(20) 


The  opposite  case  of  extremely  high  probability  events,  where  T  falls  in  the  first 
bin  is  handled  by  a  simpler  function  to  ensure  FP  drops  to  zero  by  TI=0: 


FP  =  P(T  <V  <  Xj )  +  (#  ofmembe  rs  >  T)/n  + 1 


(21) 
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where 


P(T<V<x1)  = 


vxi  J 


1 

n  + 1 


(22) 


After  combining  all  the  members  and  using  the  uniform  ranks  method  to  calculate  the  FP, 
the  resultant  output  was  an  uncalibrated  turbulence  FP  product  for  the  EUCOM  AOR. 
Figure  16  shows  the  uncalibrated  FP  for  FH  24  of  the  January  31,  2008  00Z  run. 


2008013000  FH:  024  AOR  Forecast  Probability  due  to  MDT  CAT  TURBC,  FL:  350  (MSL) 


-30  -20  -10  0  10  20  30  40 

Degrees  of  Longitude 


Figure  16.  Example  of  uncalibrated  FP  output  for  EUCOM  grid. 


5.  Validation 

To  validate  of  the  study’s  TI  and  FP  algorithm,  the  output  of  FP  of  LGT,  MDT, 
SYR  turbulence  was  visually  compared  against  the  JGE’s  upper  level  turbulence  forecast 
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for  the  same  forecast  hour  and  for  similar  flight  levels.  The  following  comparisons  made 
from  Figure  17  are  typical  results  found  when  the  researcher  compared  other  forecast 
hours  and  turbulence  intensities. 


Percent  Probability  cl  Turbulence,  FL.340  —  FL380 


Figure  17.  FH  24  of  the  January  30,  2008,  00Z  run;  valid  January  31,  2008,  00Z. 


a)  CONUS  FP  (uncalibrated)  of  MDT  turbulence  b)  JEFS  CONUS  FP  of  MDT 

turbulence 


In  area  1,  the  study’s  derived  FPs  are  larger  in  coverage  compared  to  JGE  FP.  In 
area  2,  the  derived  FP  is  higher  than  JGE  FP  by  60  to  80%.  This  difference  of  FP  in 
spatial  coverage  and  intensity  can  be  attributed  to  several  factors.  JEFS  combines  two 
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cycles  per  day  to  derive  the  forecast,  whereas  the  research  only  employs  one  cycle. 
Additionally,  the  study  used  three  levels  of  data  to  calculate  the  TI  whereas  JEFS  uses 
only  two.  Furthermore,  the  researcher  interpolated  the  data  whereas  JGE  estimated  the 
flight  levels  using  the  standard  atmospheric  heights.  Although,  the  differences  in  FP 
spatial  coverage  and  intensity  noted  are  not  of  high  concern,  because  the  overall 
similarities  generally  validate  the  processing  in  the  research.  Regardless,  the  actual  TI 
and  FP  output  is  not  of  critical  importance.  The  processes  behind  FP  and  decision  input 
calculations  are  the  main  focus. 

B.  PART  TWO 

This  part  of  the  process  involves  generating  calibrated  forecast  probabilities, 
ambiguity,  and  decision  input  (DI)  data.  The  researcher  used  a  newly  fashioned 
ambiguity  and  calibration  calculation  method  called  the  “Taijitu”  method  (pronounced 
“ti-chee-tu”).  Founded  by  F.A.  Eckel  (2008),  it  estimates  the  calibration  and  ambiguity 
(i.e.,  systematic  and  random  error)  in  the  ensemble  FP  caused  by  limited  sampling  and 
ensemble  deficiencies.  The  method  concentrates  on  the  errors  found  in  the  first  and 
second  moments  (mean  and  variance)  of  the  ensemble  PDF  since  they  have  the  greatest 
affect  on  FP  error  (Eckel,  2008). 

The  GFS  ensemble,  used  in  this  research,  is  non-ideal  (no  internal  model 
perturbations)  with  finite  sampling  (only  21  members).  GEFS  forecasts  are  likely 
underdispersive  and  highly  ambiguous.  Dispersion  is  the  ensembles  representation  of 
error  growth  (Eckel,  2007).  The  overall  (average)  underdispersion  is  the  systematic 
(predictable)  FP  error  that  can  be  corrected  for  through  calibration  (Eckel,  2008).  The 
unknown  aspect  of  FP  error  (ambiguity)  arises  from  the  random  variability  of  the  quality 
of  ensemble  dispersion  (Eckel,  2008). 

Transforming  from  a  PDF  error  to  a  FP  error  is  a  matter  of  comparing  the 
probability  of  the  ensemble  and  true  PDF  for  the  chosen  threshold.  For  example,  an 
ensemble  forecast  for  2-m  temperature  at  one  location  is  fit  by  a  Gaussian  PDF  of  A(//c  = 
1.4°C,  cre  =  0.9°C).  If  the  true  PDF  is  A(1.1°C,  1.3°C),  the  unknown  PDF  errors  are 
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0.3°C  in  jue  and  -0.4°C  in  cre.  Example  is  taken  from  Eckel  (2008).  Figure  18  shows  how 
to  find  the  error  in  FP  for  the  event  of  2-m  temperature  <  0  °C. 


True  Forecast  PDF 

- Ensemble’s  Forecast  PDF 

True  FP 
Ensemble  FP 


: ' 
CL 


Temperature  (C) 


Figure  18.  Example  comparison  of  a  true  and  an  ensemble  forecast  PDF  (upper  plot) 
and  CDF  (lower  plot).  An  error  of  -13.9%  in  forecast  probability  for  the  chance  of 
temperature  <  0°C  is  the  difference  in  the  PDF’s  shaded  areas,  or  the  difference  in  the 
two  CDFs  indicated  by  the  double  arrow  [From  Eckel,  2008], 


The  Taijitu  method  calculates  ambiguity  and  calibration  data  using  error  in  the 
first  and  second  moments  of  the  ensemble  PDF: 

1)  the  mean  of  the  error  in  the  ensemble  mean  ( juem ), 

2)  the  standard  deviation  of  the  error  in  the  ensemble  mean  ( crem ), 

3)  the  mean  of  the  error  in  ensemble  spread  ( /ues ),  and 

4)  the  standard  deviation  of  the  error  in  the  ensemble  spread  ( cres ). 
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Each  ensemble  PDF  moment  has  its  own  error  distribution  determined  by  verifying  the 
ensemble  (Figure  19),  which  is  standardized  with  respect  to  the  average  spread  of  the  true 
PDF  (Eckel,  2008).  The  standardized  error  distribution  of  the  ensemble  mean  is  assumed 
to  be  Guassian,  positioned  on  the  long-term  bias  of  the  ensemble  mean,  juem ,  with  a 

standard  deviation,  <yem  (Eckel,  2008).  The  fractional  error  distribution  for  ensemble 
spread  is  assumed  gamma. 


Forecast  Hour  84 


Forecast  Hour  84 


Figure  19.  Sample  standardized  error  distributions  for  FH  84. 


In  this  research,  ensemble  mean  bias  ( juem )  was  set  to  zero  following  the 
assumption  that  the  TI  thresholds  were  calibrated  to  account  for  any  TI  bias.  The 
standardized  <jem  was  determined  using  a  simplified  process.  Rather  than  computing  the 

f 

variance  of  error  in  ensemble  mean  from  all  the  data  subsets,  the  researcher  used  the  oe 
distribution  whose  variance  is  dependant  on  ensemble  size  (Eckel,  2008).  Standardized 
to  a  normal  PDF  where  N(0,1),  oem  =1  /-Jn  (where  n=21  members)  «  0.218  (Eckel, 
2008) . 

To  find  the  distribution  of  fractional  error  in  ensemble  spread,  the  researcher 
acquired  a  long-term  verification  dataset  of  GEFS  for  turbulence  related  parameters  from 
AFWA.  Ideally,  the  error  distribution  would  be  for  TI  itself,  but  this  is  not  practical  since 
TI  observations  are  not  available.  The  researcher  used  the  GEFS  geopotential  height 
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output  at  the  500  hPa  height  level  (H5)  to  estimate  the  ensemble  TI  error  since  1)  winds 
are  generally  geostrophic  at  this  level  in  the  middle  to  high  latitudes  and  2)  the  spacing  of 
the  geopotential  heights  indicate  the  speed  of  the  wind  with  their  inversely  proportional 
relationship  (Wallace  and  Hobbs,  1977).  Since  the  TI  calculation  is  based  on  vertical  and 
horizontal  wind  shear,  one  can  assume  the  tighter  (looser)  the  geopotential  height 
gradient  (vertically  and  horizontally)  is,  the  greater  (lesser)  the  wind  shear  is.  Hence,  the 
ensemble  error  in  the  geopotential  heights  will  reflect  in  the  ensemble  error  in  TI. 

The  long-term  H5  spread  variance  data  contains  total  variance  and  total  mean 
squared  error  of  the  ensemble  mean  (MSE)  information  for  the  30-60°N  global  latitude 
band  (10,800  total  grid  points).  The  data  spans  from  August  25  to  September  29  (37  days 
total)  for  the  00Z  model  run.  The  researcher  trimmed  the  data  set  to  20  days  (by  taking 
every  other  two  days)  to  limit  correlation  among  results.  Twenty  days  worth  of  data  was 
deemed  sufficient  to  estimate  error  variability.  From  this  20  day  set,  approximately  five 
percent  of  the  data  is  missing  since  many  days  have  one  or  more  forecast  hours  (tau) 
missing.  The  researcher  sorted  the  20-day  dataset  by  tau  and  calculated  the  fractional 
error  in  the  spread  (FES)  for  each  day: 

FES  =  — — — 

RMSE-  (23) 

where  ae  is  the  average  ensemble  spread  and  RMSE-  is  the  root  mean  square  error  of  the 
ensemble  mean.  The  following  table  shows  the  daily  FES  values  for  forecast  hour  84. 
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DATE 

FES  (FH  84) 

2007082500 

0.719 

2007082800 

0.866 

2007082900 

0.792 

2007090200 

0.821 

2007090300 

0.799 

2007090500 

0.716 

2007090600 

0.799 

2007090900 

0.841 

2007091000 

0.877 

2007091200 

0.793 

2007091300 

0.774 

2007091400 

0.632 

2007091600 

0.899 

2007091800 

0.673 

2007092000 

0.784 

2007092300 

0.737 

2007092500 

0.803 

2007092600 

0.709 

2007092800 

0.812 

2007092900 

1.02 

/A 

0.793 

0.0859 

Table  2.  All  Fractional  Error  Spread  (FES)  values  for  forecast  hour  (FH)  84. 


To  find  the  mean  of  the  error  in  ensemble  spread  (//es),  all  the  daily  FES 
computations  were  averaged  for  one  tau.  Averaging  all  the  FES  for  FH  84  yields  /Jes  = 
0.793.  The  /ues  is  the  systematic  aspect  of  the  spread  error  which  can  be  corrected  for 
through  calibration  (Eckel,  2008).  To  calculate  the  random  aspect  of  spread  error  ( aes ), 
the  standard  deviation  of  the  20  FES  values  was  computed.  The  aes  represents  the 
random  aspect  of  the  ensemble  spread  error  (Eckel,  2008),  which  creates  ambiguity.  The 
standard  deviation  for  FH  84  FES  data  yielded  cres  =  0.0859. 

Figure  20a  illustrates  the  effect  on  FP  error  for  10  random  draws  from  the 


ensemble  mean  and  spread  error  distributions.  The  standardized  error  distribution  of  the 
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ensemble  mean  represents  the  ensemble  PDF  location  error,  while  the  fractional  error 
distribution  for  ensemble  spread  represents  the  ensemble  PDF  width  error  (with  respect  to 
the  true  PDF).  Sample  1  (Figure  20a)  is  driven  by  primarily  location  error.  Sample  2  is 
driven  by  primarily  width  error.  Variation  of  samples  1  and  2  are  driven  by  a 
combination  of  location  and  width  errors. 

Taking  many  random  samples  (1000)  yields  a  distribution  of  potential  error  in 
FPor  ambiguity  (Figure  20b).  The  results  for  1000  trials  produced  a  skewed  distribution 
which  can  be  fitted  by  a  beta  PDF.  Figure  21  displays  example  histograms  of  FP  errors 
resulting  from  the  FH  84  ensemble  PDF  error  distributions  (shown  in  Figure  19).  The 
distribution  of  possible  FP  error  skew  away  from  the  worst  error;  the  direction  of  the 
skew  is  dependant  on  the  FP.  Using  Figure  21  as  an  example,  the  beta  distribution  for 
FP=25%  skewed  to  left  (Figure  21a)  and  FP=75%  skewed  to  right  (Figure  21b). 

The  5th,  50th,  95th  percentiles  of  the  beta  PDFs  for  all  FP  values  are  plotted  in 
Figure  22.  The  50th  percentile  the  beta  PDF  (Figure  21)  equates  to  the  mean  of  the 
ensemble  error.  Because  the  mean  of  the  ensemble  error  represents  a  systematic  error, 
calibration  can  be  accomplished  by  adding  the  50th  percentile  values  to  the  raw  FP.  Thus 
‘calibration’  is  the  reverse  of  the  50th  percentile  curve.  In  a  90  percent  confidence 
interval  (Cl),  the  5th  and  95th  percentile  of  the  beta  PDF  equate  to  the  spread  of  the 
ensemble  error  (Eckel,  2008)  and  represent  the  variability  of  the  FP  error  (random  error). 
The  non-calibrated  50th  percentile  is  subtracted  from  95th  percentile  and  the  5th  percentile 
is  subtracted  from  the  non-calibrated  50th  percentile  to  achieve  the  upper  and  lower 
ambiguity  bounds  respectively.  The  ambiguity  bounds  are  added  to  the  calibrated  50th 
percentile  curve  to  compose  the  “Taijitu”  curve  pictured  in  Figure  23.  The  method  takes 
it  name  from  the  resemblance  it  has  to  the  Chinese  taijitu  symbol  (Eckel,  2008). 
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a) 


Possible  Error  in  Forecast  Probabilty  given  FP  =  0.25 


b) 


Possible  Error  in  Forecast  Probabilty  given  FP  =  0.75 


Figure  21 .  Histogram  of  1000  sample  FP  errors,  with  fitted  beta  distribution,  resulting 
from  the  ensemble  PDF  error  distributions  in  Figure  19  for  FP  of  (a)  25.0%,  (b)  75.0% 
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Ensemble  Forecast  Probability 


Figure  22.  Uncalibrated  5th,  50th,  95th  percentiles  of  the  beta  PDFs  of  possible  FP 

error  for  FH  84. 


Calibrated  Ambiguity  Curve  for  FH:  84,  Cl:  0.9 


Figure  23.  Ambiguity  and  Calibration  Curve  for  FH  84,  90%  Cl:  The  FP  calibration 
(thick  line)  is  simply  the  opposite  of  the  50th  percentile  of  the  ensemble  error  distribution 
(beta  PDF).  The  upper  and  lower  bounds  of  ambiguity  (thin  lines)  are  the  90% 
confidence  interval  of  the  true  forecast  probability,  found  by  taking  the  difference  of  the 
5th  and  95th  percentiles  from  the  50th  percentile.  Displayed  is  calibration  factor  for 
FP=10%,  upper  and  lower  ambiguity  bounds  for  calibrated  FP  (16%)  [After  Eckel,  2008]. 
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To  formulate  the  error  bars,  the  calibration  factor  is  added  to  the  uncalibrated  FP. 
The  ambiguity  intervals  are  computed  from  the  uncalibrated  FP.  For  example  in  Figure 
23,  the  upper  bound  of  the  ambiguity  interval  is  +12%  and  the  lower  bound  is  -8%  for 
FP=10%.  The  calibration  factor  for  FP=10%  is  +6%,  which  makes  the  calibrated 
FP=16%.  Hence,  ambiguity  interval  for  FP=16%  is  8-28%.  Note,  because  of  the  “s” 
shaped  curved,  most  upper  and  lower  bounds  of  the  ambiguity  intervals  are  not  equal. 

In  order  to  communicate  optimized  decision  input,  the  proper  confidence  interval 
(for  the  beta  PDF  described  above)  for  the  ambiguity  intervals  must  be  chosen.  The 
amber  regions  in  the  research  decision  input  products  are  influenced  by  the  extent  of  the 
ambiguity  intervals.  If  the  ambiguity  intervals  are  too  small,  the  amber  regions  will  be 
smaller,  giving  the  user  a  false  sense  of  high  certainty  in  the  decision  input.  Conversely, 
if  the  ambiguity  intervals  are  too  large,  the  amber  regions  might  be  too  large,  which 
removes  the  ability  of  the  product  to  give  decisive  recommendations. 

The  researcher  tested  CIs  (70%,  80%,  90%,  95%,  and  99%)  to  determine  the 
affect  on  the  DIs.  The  higher  (lower)  Cl  conveyed  more  (less)  of  the  ambiguity.  Figure 
24  illustrates  the  maximum  ambiguity  bounds  for  each  Cl  at  FH  84.  The  Cl  with  average 
maximum  bound  (approximately  ±  1 5  %)  is  the  90%  Cl.  This  Cl  is  a  balance  between  the 
70%  and  99%  CL  It  reveals  a  large  degree  of  the  calculated  ambiguity,  but  still  yields  a 
reasonable  decision  aid.  For  these  reasons,  the  90%  Cl  is  the  chosen  Cl  for  this  study.  See 
Appendix  A  for  three  forecast  hour  comparisons  of  varying  CI. 

To  subjectively  approximate  uncertainty  (error  bars)  for  RT,  the  researcher 
assumed  a  +  5  %  confidence  interval  to  represent  the  user’s  inability  to  specify  a  precise 
RT.  Using  this  confidence  interval  for  all  RT  is  not  realistic.  A  user  with  a  RT  <0.05 
would  have  a  negative  RT  interval  bound.  A  user  with  a  RT  >0.95  will  have  a  RT  interval 
bound  >100%.  The  researcher  used  a  second  order  polynomial  curve  to  avoid  this 
problem  achieving  the  RT  uncertainty  intervals  (Figure  25).  As  with  FP,  users  with 
extreme  high  and  low  RT  (i.e.  RT=0.00  or  1.0)  have  the  most  confidence  of  their 
tolerance  to  risk  (i.e.  absolute  tolerance  to  no  or  all  risk).  The  RT=0.5  user  may  be  unsure 
of  his/her  tolerance  to  risk.  Hence,  RT=0.5  has  the  most  uncertainty  with  interval  bounds 
of  ±0.05. 
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To  incorporate  the  ambiguity  and  calibration  data  into  the  graphical  products,  the 
data  cycled  through  a  DI  algorithm  to  compare  FP  and  RT,  and  produce  stop-light  DI 
data.  Using  look-up  tables,  the  researcher  added  the  calibrated  factor  to  each  of  the  grid’s 
FP.  The  ambiguity  bounds  were  added  to  the  calibrated  FP  to  formulate  the  ambiguity 
interval  about  FP.  To  formulate  the  stoplight  DI  product,  the  algorithm  compared  the 
confidence  intervals  of  FP  and  RT.  If  the  lower  bound  of  the  FP  error  bar  is  above  the 
upper  bound  of  the  RT  error  bar,  then  the  risk  is  deemed  unacceptable  (no-go 
recommendation)  and  the  grid  point  is  colored  red  (Figure  26a).  If  lower  bound  of  the 
RT  error  bar  is  above  the  upper  bound  of  the  FP  error  bar,  then  risk  is  deemed  acceptable 
(go  recommendation)  and  the  grid  point  is  colored  green  (Figure  26b).  Otherwise,  the  FP 
and  RT  error  bars  overlap,  so  there  is  no  clear  decision  (inconclusive)  and  the  grid  point 
is  colored  amber  (Figure  26c).  The  minimum  confidence  interval  in  the  green  and  red  DI 
(the  FP  and  RT  probability  are  independent  of  one  another)  is  (Cl  +  (100-CI)/2)2.  The 
minimum  Cl  occurs  when  the  FP  and  RT  error  bars  are  stacked  atop  one  another  (the  end 
bounds  just  meet  one  another).  With  the  research  0=90%,  the  minimum  confidence  is 
952%  =90.25%. 

As  stated  previous,  using  CIs  with  different  significance  alters  the  length  of  the 
FP  error  bars  (ambiguity  intervals),  therefore  increasing  or  decreasing  the  amount  of 
ambiguity  conveyed.  Inconclusive  DI  is  risk  that  is  too  close  to  call;  either  a  go  or  no-go 
DI  may  be  acceptable.  In  addition  to  the  calibrated  FP  and  stoplight  DI  graphical  product, 
an  additional  product  displays  the  maximum  or  “worst-case”  potential  error  (ambiguity) 
in  the  FP.  In  order  to  generate  this  product,  lower  and  upper  bounds  of  the  FP  ambiguity 
intervals  are  compared  with  each  other  at  each  grid  point.  Which  ever  bound  is  greater  is 
deemed  as  the  worst-case  potential  error  (PE). 
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Figure  24.  Ambiguity  Curves  for  FH  84,  CIs  a)  70%  b)  80%  c)  90%  d)95%  e)  99% 


Risk  Tolerance  Uncertainty  Curve,  90%  Cl 


Figure  25.  Risk  Tolerance  Uncertainty  Curve. 


C.  PART  THREE 

In  order  to  visualize  DIs  from  Part  two,  the  researcher  devised  a  graphical  user 
interface  (GUI).  If  all  the  possible  graphical  products  were  generated  for  one  risk 
tolerance  (13  flight  levels,  14  forecast  hours,  and  3  turbulence  intensities),  the  total 
number  of  products  would  be  546.  Using  a  simple  GUI,  a  user  can  generate  one  set  of 
graphical  products  for  his/her  specific  risk  tolerance,  mission  threshold  turbulence 
intensity,  forecast  hour,  and  flight  level  (Figure  27)  and  eliminate  unnecessary  product 
generation. 


53 


a) 


0.55 

-  RT:  0.5 

0.45 


0.69 

FP:  0.67 

0.63 


b) 


Decision  Input:  Risk  Unacceptable 


0.55 

-  RT:  0.5 

0.45 


0.38 

-  FP:  0.33 
0.31 


Decision  Input:  Risk  Acceptable 


c) 

0.62 

0.55  :  FP:0-56 

—  RT:  0.5 

0.48 

0.45 

Decision  Input:  Inconclusive 


Figure  26.  (a-c)  Calibrated  FP  and  user  RT  (center  of  error  bar)  and  associated 

confidence  intervals  showing  different  DIs 
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Figure  27.  Prototype  GUI. 


In  the  first  block,  the  user  has  the  option  of  either  using  the  drop  down  menu  or 
inputting  a  specific  risk  tolerance.  The  first  option  is  geared  toward  those  users  that  have 
not  conducted  a  thorough  study  of  their  risk  tolerance  and/or  are  not  educated  in 
stochastic  methods  and  decision  science  (a  beginner  user).  The  drop  down  menu  includes 
generic  risk  tolerances  and  a  description/example  for  the  specific  risk  tolerance.  In 
parenthesis,  the  relative  frequency  is  displayed  for  the  number  of  times  out  of  ten  the  user 
requires  no  weather  (WX)  impact  (Figure  28a).  Research  has  shown  using  natural 
frequencies  helps  users  make  better  statistical  inferences  and  interpret  the  data  more 
correctly  than  probabilities  (Hoffrage  et  al,  2000).  It  is  important  to  note,  the  users 
requirement  to  have  no  weather  impact  is  only  valid  within  potential  areas  of  turbulence, 
not  outside.  In  the  second  through  fourth  blocks,  the  user  chooses  his/her  mission  limiting 
turbulence  intensity  level  (Figure  28),  forecast  hour  (Figure  29a),  and  flight  level  (MSL) 
(Figure  29b). 
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After  submitting  all  the  inputs,  the  end  result  is  the  generation  of  three  types  of 
graphical  products:  calibrated  FP,  maximum  PE  in  FP  and,  a  stop-light  DI  product. 
Ideally,  the  FP  and  the  PE  data  should  be  overlaid,  but  Matlab  is  too  limited.  Showing  all 
these  products  together  furthers  education  and  gives  validity  to  the  DI  product.  It  helps 
the  uneducated  user  understand  the  process  behind  the  final  DI  product.  The  examples  in 
Figure  30  thru  Figure  32  uses  mission  threshold  turbulence  intensity  of  MDT,  forecast 
hour  of  24,  flight  level  of  35,000  feet  (MSL),  and  a  RT=0.3  or  30%.  Instead  of 
displaying  FP  for  the  chance  of  turbulence,  the  scale  was  reversed  to  show  mission 
success  chance  due  to  turbulence  for  more  operational  relevance  (i.e.  probability  of  no 
impact  from  MDT  turbulence),  shown  in  Figure  30. 

To  illustrate,  points  A,  B,  C  in  Figure  30-Figure  32  represent  three  different 
decision  inputs.  Point  A  represents  10%  chance  of  mission  success  (90%  FP)  in  Figure 
30.  The  worst  case  PE  at  this  point  is  approximately  8%  (Figure  31),  so  the  maximum 
ambiguity  interval  is  from  82-98%.  The  uncertainty  interval  for  RT=30%  (according  to 
Figure  25)  is  +  4%.  With  RT  interval  of  26-34%  <  FP=82-98%,  the  DI  for  point  A  is 
unacceptable  risk  (red)  shown  in  Figure  32.  Point  B  represents  an  approximately  60% 
chance  of  mission  success  (40%  FP).  The  worst  case  PE  at  this  point  is  approximately 
12%,  so  the  maximum  ambiguity  interval  is  from  28-52%.  Since  this  interval  overlaps  the 
RT  interval  of  26-34%  by  6%,  point  B  is  deemed  inconclusive  (amber).  Point  C 
represents  a  90%  chance  of  mission  success  (10%  FP).  The  worst  case  PE  at  this  point  is 
approximately  7%,  so  the  maximum  ambiguity  interval  is  3-17%.  With  RT  interval  of  26- 
34%  >  FP=3-17%,  the  DI  for  point  C  is  acceptable  risk  (green)  shown  in  Figure  32. 
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a) 


b) 


Figure  28.  Prototype  GUI:  Drop  down  menu  of  a)  generic  risk  tolerances  b)  mission 

limiting  turbulence  intensities 
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a) 


b) 


Figure  29.  Prototype  GUI:  a)  Drop  down  menu  of  forecast  hours  b)  Drop  down  menu 

of  flight  levels 
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2008013000  FH:  024  Mission  Success  Chance  {%)  due  (o  MDT  CAT  TURBC,  FL:  350  (MSL) 
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-30  -20  -10  0  10  20  30  40 
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Figure  30.  EUCOM  Mission  Success  Chance  (FP  calibrated)  due  to  MDT  turbulence 
for  FH  24  of  the  January  30,  2008,  00Z  run;  valid  January  31,  2008,  00Z 


2008013000  FH:  024  Worst  Case  Potential  Error  {*/-)  For  FP  of  MDT  CAT  TUFtBC,  FL:  350  (MSL) 


Longitude 

Figure  3 1 .  EUCOM  Worst  Case  Potential  FP  Error  of  MDT  turbulence  for  FH  24  of 
the  January  30,  2008,  00Z  run;  valid  January  31,  2008,  00Z 
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2008013000  FH:  024  Decicision  Input  for  RT:  0.3  or  MDT  CAT  TURBC,  FL:  350  (MSL) 


Longitude 


Risk  Acceptable 


Inconclusive 

□ 


Risk  Unacceptable 


Figure  32.  EUCOM  DI  for  RT :  0.3  of  MDT  turbulence  for  FH  24  of  the  January  30, 

2008,  00Z  run;  valid  January  31,  2008,  00Z 
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IV.  RESULTS 


A.  EVOLUTION  OF  TAIJITU  CURVES  THROUGH  TIME  WITH 
INCREASING  LEAD  TIME 

Table  3  shows  the  mean  and  standard  deviation  results  for  all  forecast  hours  for 
the  distribution  of  the  EF  spread  fractional  error.  The  general  reduction  of  mean  in  the  EF 
spread  is  due  to  dispersion  deficiencies  in  the  GEFS.  Early  in  the  forecast  period, 
FH  00  -  18,  the  magnitude  of  jues  was  greater  than  one.  At  FH  24,  the  magnitude  was 

approximately  equal  to  one.  Beyond  FH  24,  the  magnitude  of  jues  dropped  below  one. 

This  trend  is  due  to  the  ensemble  compensating  for  its  underdispersion.  As  stated  in  the 
previous  chapter,  GEFS  is  underdispersive  because  of  limited  sampling  and  lack  internal 
model  perturbations.  To  somewhat  compensate  for  the  underdispersion,  GEFS  over 
perturbs  its  initial  conditions.  The  actual  error  growth  exceeded  the  ensemble  dispersion, 
so  by  FH  24,  the  magnitude  of  jues  <  1 .  When  compared  to  the  true  PDF,  the  ensemble 

PDF  was  too  wide  early  in  the  forecast  period,  turning  too  narrow  later  in  the  forecast 
period.  This  reverse  in  of  the  calibration  curve  through  the  forecast  period  (Figure  33) 
was  in  response  to  the  change  in  average  ensemble  error  from  >1  to  <1  (driven  by  the 
downward  trend  of  the  ensemble  spread  errors).  The  calibration  curve  corrected  the 
average  error  as  explained  in  Chapter  II. 

Note  the  additional  downward  trend  of  the  values  for  the  standard  deviation  of 
error  in  EF  spread,  aes  (Table  3).  As  forecast  lead-time  increased,  the  ensemble  members 

diverged  toward  the  climatological  or  climatic  variance,  the  maximum  limit  of  the 
ensemble  spread.  The  climatic  variance  is  the  variance  of  all  naturally  occurring  states 
(Eckel,  2007).  As  the  ensemble  spread  reached  this  maximum  spread,  all  the  members 
asymptoted  to  this  limit.  Hence,  the  standard  deviation  of  the  error  in  EF  spread 
decreased. 
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Even  though  the  standard  deviation  of  the  ensemble  spread  error  aes  generally 
decreased,  the  ambiguity  intervals  remained  almost  constant.  Recall  that  <iem  is 

dependant  only  on  ensemble  size  and  is  fixed  at  1  /  4n  (where  n=21  members) 
«  0.218  (Eckel,  2008).  The  nearly  constant  ambiguity  bounds  for  each  Cl  reveal  that  crem 
dominated  the  random  FP  error  in  comparison  to  the  contribution  from  decreasing  cres . 


Forecast  Hour 

Mean  of 

Standard  Deviation  of 

Error  in  EF  spread,  /ues 

Error  in  EF  Spread,  cres 

00 

1.19 

0.154 

06 

1.28 

0.179 

12 

1.14 

0.147 

18 

1.12 

0.131 

24 

1.0077 

0.125 

30 

0.985 

0.115 

36 

0.921 

0.112 

42 

0.9102 

0.0998 

48 

0.862 

0.1046 

54 

0.857 

0.098 

60 

0.845 

0.0755 

66 

0.838 

0.0788 

72 

0.821 

0.0774 

78 

0.816 

0.0818 

84 

0.793 

0.0859 

Table  3.  Mean  and  standard  deviation  values  for  the  EF  spread  fractional  error 

distribution 
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Calibrated  Ambiguity  Curve  for  FH:  84,  Cl:  0.8 


Figure  33  continued. 


B.  ANALYSIS  OF  DECISION  INPUT  (DI)  WITH  ADDED  UNCERTAINTY 

INFORMATION 

1.  Simulated  DI  Product  and  Analysis  Construction 

In  order  to  show  potential  value  added  and  highlight  differences  between  current 
decision  input  (DI)  products  and  research  results  (RSCH),  the  researcher  constructed 
products  focused  only  on  the  risk  of  MDT  turbulence  to  simulate  IWEDA  and  WRAP  to 
compare  with  RSCH.  The  unfavorable  (mission)  threshold  of  MDT  turbulence  used  for 
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WRAP  and  I  WED  A  was  TI  >  5.  The  marginal  threshold  for  I  WED  A  and  WRAP  was 
assumed  to  be  half  the  magnitude  between  the  MDT  TI  threshold  and  LGT  TI  threshold 
(=2),  which  equates  to  TI  >  3.5. 

To  create  a  baseline  deterministic  product,  all  TI  values  in  the  control  member  >  5 
equated  to  a  forecast  of  MDT  turbulence.  The  product  implies  either  zero  or  one  hundred 
percent  probability  of  turbulence  (Figure  34). 


2007100200  FH:  084  Deterministic  Forecast  (cOO)  MDT  or  greater  CAT  TURBO,  FL:  350  (MSL) 


Figure  34.  Deterministic  MDT  turbulence  forecast  using  control  member  for  the  fall 

run,  FH:  84,  FL:  350 


To  simulate  IWEDA,  the  researcher  applied  the  marginal  and  mission  MDT 
turbulence  thresholds  to  the  control  member’s  TI.  The  researcher  designated  these 
specific  values  to  each  result  in  order  to  link  them  to  the  stop-light  colors  on  the  Matlab 
colorbar.  If  the  TI  for  the  grid  point  exceeded  the  marginal  or  mission  threshold,  then  the 
grid  point  was  set  to  the  value  of  1.5  (colored  amber)  or  3.0  (colored  red)  respectively.  If 
neither  of  the  thresholds  are  exceeded,  the  grid  point  was  set  to  a  value  of  zero  (colored 
green).  The  green,  amber,  and  red  colors  represent  the  decision  inputs  of  acceptable  risk, 
inconclusive,  and  unacceptable  risk  respectively  (Table  4). 


66 


TI  threshold 

Color  Value 

Color  Assignment 

Decision  Input 

<3.5 

0.0 

Green 

Acceptable  Risk 

>3.5  (marginal) 

1.5 

Amber 

Inconclusive 

>5.0  (mission) 

3.0 

Unacceptable  Risk 

Table  4.  IWEDA  Simulation:  TI  Thresholds  and  corresponding  color  values,  color 

assignments,  and  DI 


To  simulate  WRAP,  the  researcher  compared  the  FP  of  exceeding  the  IWEDA 
marginal  and  mission  thresholds  to  a  fixed  RT  to  assign  the  color  values.  If  the  FP  for 
exceeding  the  mission  threshold  is  greater  than  the  RT,  then  the  grid  point  was  set  to  3.0 
(red).  Otherwise,  if  the  FP  for  exceeding  the  marginal  threshold  is  greater  than  the  RT, 
then  the  grid  point  was  set  to  1.5  (amber).  Otherwise,  the  grid  point  was  set  to  0.0  (green) 
since  neither  FP  exceeds  the  RT.  See  Table  5. 


FP  vs  RT 

Color  Value 

Color  Assignment 

Decision  Input 

FP  (  of  either  threshold) 

<  RT 

0.0 

Green 

Acceptable  Risk 

Marginal  FP  >  RT 

1.5 

Amber 

Inconclusive 

Mission  FP  >  RT 

3.0 

Unacceptable  Risk 

Table  5.  WRAP  Simulation:  FP  vs  RT  and  corresponding  color  values,  color 

assignments,  and  DI. 


To  show  differences  between  the  IWEDA-like,  WRAP-like,  and  RSCH  derived 
DI  products,  these  color  values  were  subtracted  in  the  following  manner:  IWEDA  minus 
WRAP;  WRAP  minus  RSCH.  To  illustrate,  when  the  subtraction  of  WRAP  from 
IWEDA  yielded  positive  three,  IWEDA  forecasted  red  conditions  and  WRAP  forecasted 
green  conditions,  translating  to  a  high  (pessimistic)  decision  input  bias  of  IWEDA 
relative  to  WRAP.  Conversely,  when  the  subtraction  yielded  negative  -1.5,  this  means 
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IWEDA  forecasted  green  or  amber  conditions,  and  WRAP  forecasted  either  yellow  or  red 
conditions  respectively,  which  translated  to  IWEDA  having  a  low  (optimistic)  decision 
input  bias.  Grey  shading  means  equal  decision  inputs.  Table  6  displays  the  DI  bias 
colorbar  key  for  IWEDA  minus  WRAP  DI  and  WRAP  minus  RSCH  DI  products. 


DECISION  INPUT  (DI)  DIFFERENCE  (BIAS)  COLORBAR  KEY: 


OPTIMISTIC  BIAS 


Dark  Green 
(color  value 
difference:  -3) 


IWEDA- WRAP 
OUTPUT: 


Green 
(color  value 
difference:  -1.5) 


IWEDA- WRAP 
or 

WRAP-RSCH 

OUTPUT: 


ZERO  BIAS 


Gray 

(color  value 
difference:  0) 


IWEDA- WRAP 
or 

WRAP-RSCH 

OUTPUT: 


□  - 
□  - 


PESSIMISTIC  BIAS 


IWEDA- WRAP 
or 

WRAP-RSCH 

OUTPUT: 


IWEDA- WRAP 
OUTPUT: 


Table  6.  DI  Bias  Colorbar  Key.  Note:  WRAP  and  RSCH  never  differ  by  more  than 

one  DI  category. 


Using  the  two  cases  of  0000  UTC,  2  October  2007  (fall)  run  and  the  0000  UTC 
30  January  2008  (winter)  run,  the  researcher  examined  MDT  turbulence  decision  input 
differences  using  FH:  06  and  84  and  RT:  0.1,  0.5,  0.9.  The  fall  case  results  are  discussed 
in  this  chapter;  the  winter  case  resultant  plots  are  in  Appendix  B  and  C. 
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2. 


Simulated  IWEDA  versus  Simulated  WRAP 


IWEDA  and  WRAP  use  the  same  marginal  and  mission  thresholds,  but  employ 
the  thresholds  through  different  methods  to  assume  a  DI.  IWEDA  depends  on  a  single 
forecast  whereas  WRAP  incorporates  21  forecast  and  user’s  risk  tolerance  information. 
The  decision  inputs  are  sometimes  vastly  different. 

a.  Fall  Case:  FH  06 

To  give  a  general  comparison,  the  FP  (stochastic)  product  conveys 
uncertainty  information,  while  the  deterministic  product  gives  no  uncertainty 
information.  The  deterministic  chart  implies  one  hundred  percent  chance  of  MDT 
turbulence  inside  the  forecasted  areas  and  zero  percent  chance  outside  the  forecasted 
areas  (Figure  35a).  The  FP  product  shows  the  spread  amongst  the  ensemble’s  members 
with  respect  to  the  event  (Figure  35b). 

Area  1  (Figure  35)  on  the  FP  chart  exhibited  40-60%  probability  of  MDT 
turbulence,  since  some  the  members  forecast  TI  >  5.  The  same  region  of  area  1  in  the 
deterministic  forecast  gave  no  hint  to  turbulence  since  the  TI  was  below  five  in  the 
control  member.  A  user  with  a  low  risk  tolerance  needed  the  probability  information  to 
incorporate  this  into  their  mission  planning.  The  potential  loss  for  encountering  moderate 
turbulence  was  much  too  high  for  a  low  risk  tolerant  user,  so  it  is  more  beneficial  if  the 
user  did  not  proceed  with  the  mission. 

Conversely,  when  the  deterministic  product  gives  a  100%  chance  of 
moderate  turbulence  (as  in  area  2  of  Figure  35),  no  user  with  a  mission  threshold  of  MDT 
turbulence  can  operate.  Using  the  FP  product,  the  user  can  see  the  FP  was  not  100%  but 
only  50-80%.  Users  with  high  risk  tolerances  are  able  to  operate  because  they  can  afford 
the  potential  loss. 
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2007100200  FH:  006  Deterministic  Forecast  (cOO)  MDT  or  greater  CAT  TURBC,  FL:  350  (MSL) 


2007100200  FH:  006  Forecast  Probability  or  MDT  CAT  TURBC,  FL:  350  (MSL) 


-30  -20  -10  0  10  20  30  40 


Longitude 

Figure  35.  Fall  run,  FH  06,  FL  350  MSL.  a)  Deterministic  MDT  turbulence  forecast 
b)  Stochastic  MDT  turbulence  forecast  using  FP 

The  differences  between  the  IWEDA  and  WRAP,  with  RT=0.5,  are 
minimal  (Figure  36).  The  forecast  evolution  is  only  six  hours.  The  members  of  the 
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ensemble  have  not  begun  to  spread  widely  at  this  point,  so  for  the  majority  of  the  region, 
high  certainty  exist  (many  areas  with  zero  or  100  percent  probability).  So,  in  a  way,  the 
stochastic  forecast  is  acting  like  the  deterministic  forecast.  Note,  the  same  areas  marking 
100%  probability  of  turbulence  in  the  deterministic  product  (Figure  35a)  are  the  same 
areas  deemed  as  unacceptable  risk  (red)  in  the  IWEDA  product  (Figure  37).  Similarly, 
the  areas  marking  >50%  probability  of  turbulence  in  stochastic  product  (Figure  35b)  are 
the  same  areas  deemed  as  red  in  the  WRAP  product  (Figure  38). 


The  IWEDA- WRAP  DI  difference  plot  is  a  mixture  of  optimistic  and 


pessimistic  biases  (Figure  36).  The  mixture  is  due  to  WRAP’S  RT=0.5,  designed  to 
emulate  a  deterministic  decision  maker.  The  RT=0.5  forces  the  stochastic  DI  to  act  like  a 
deterministic  forecast  where  the  recommendation  is  binary  (go  or  no-go).  As  stated  in  the 
previous  section,  the  risk  is  inconclusive  (amber)  for  any  FP  for  exceeding  the  marginal 
threshold  >  RT  and  unacceptable  (red)  for  any  FP  exceeding  the  mission  threshold  >  RT. 


2007100200  FH:  006  DI  BIAS:  IWEDA  minus  WRAP  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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No  Bia3 


IWEDA  Pessimistic  Bias 


Figure  36.  Fall  run,  FH  06,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0.5 
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In  regions  of  area  1  (Figure  37),  IWEDA  forecasted  green  because  the 
single  TI  solution  <3.5.  WRAP  forecasted  amber  for  the  majority  of  area  1  because  the 
FP  for  exceeding  the  marginal  threshold  >  0.5  (Figure  38). 


2007100200  FH:  006  Deterministic  (IWEDA-like)  Dl  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


-30  -20  -10  0  10  20  30  40 

Longitude 


Risk  Acceptable  Inconclusive  Risk  Unacceptable 

if  ■  ■ 

Figure  37.  Fall  run,  FH  06,  FL  350  MSL.  Simulated  IWEDA  MDT  turbulence  DI 

Lowering  the  RT  to  0.1  for  WRAP  changed  the  DI  dramatically  (Figure 

39).  Because  the  user  can  not  withstand  much  risk,  (FP)  >0.1,  many  of  the  areas 

previously  deemed  acceptable  or  inconclusive  were  deemed  unacceptable.  Users  with 

low  risk  tolerances  can  not  afford  to  take  large  risks  because  potential  losses  are 

relatively  too  large,  and  the  cost  to  protect  is  proportionally  smaller.  IWEDA  DI 

remained  the  same.  With  predominantly  more  red  and  amber  regions  in  WRAP  DI,  the 

IWEDA  DI  bias  was  all  optimistic  (Figure  40).  Using  IWEDA  DI,  a  low  risk  tolerant 

user  flying  into  area  1  was  given  a  false  sense  of  confidence  that  the  mission  will  likely 

not  be  overcome  by  moderate  turbulence  in  area  2  (Figure  37).  WRAP  revealed  the 

mission  can  not  afford  to  take  the  risk  operating  in  area  2  in  Figure  39. 
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2007100200  FH:  006  Stochastic  (WRAP-like)  Dl  forRT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


-30  -20  -10  0  10  20  30  40 

Longitude 


Risk  Acceptable 


ccept 


Inconclusive  Risk  Unacceptable 


□ 


Figure  38.  Fall  run,  FH  06,  FL  350  MSL.  Simulated  WRAP  MDT  turbulence  DI  with 

RT=0.5 


Increasing  the  RT  to  0.9  for  WRAP  again  changed  the  DI  dramatically 
(Figure  41).  A  user  with  a  risk  tolerance  of  0.9  can  withstand  the  large  risk;  their 
potential  losses  are  relatively  small,  almost  equal  to  their  cost  to  protect.  Many  of  the 
areas  previously  deemed  risk  unacceptable/inconclusive  were  deemed  risk  acceptable. 
With  the  predominantly  more  green  regions  forecasted  for  WRAP,  the  IWEDA  DI  bias 
was  all  pessimistic  (Figure  42).  Using  the  IWEDA  DI,  a  high  risk  tolerant  user  expected 
major  impacts  in  parts  of  the  area  3  (Figure  37).  Using  the  WRAP  DI,  most  of  area  3 
(Figure  41)  is  instead  deemed  inconclusive  (amber)  because  of  WRAP’S  inclusion  of  FP 
and  RT. 
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2007100200  FH:  006  Stochastic  (WRAP-like)  Dl  for  RT:  0.1  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


0  10 
Longitude 

Inconclusive 


20  30  40 

Risk  Unacceptable 


Risk  Acceptable 

□ 


Figure  39.  Fall  run,  FH  06,  FL  350  MSL.  Simulated  WRAP  MDT  turbulence  DI  with 

RT=0.1 


2007100200  FH:  006  Dl  BIAS:  IWEDA  minus  WRAP  Tor  RT:  0.1  oF  MDT  CAT  TURBC,  FL:  350  (MSL) 


IWEDA  Optimistic  Bias  No  Bias  IWEDA  Pessimistic  Bias 


Figure  40.  Fall  run,  FH  06,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0. 1 
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Figure  41. 


Figure  42. 


2007100200  FH:  006  Stochastic  (WRAP-like)  Dl  Tor  RT:  0.9  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


Lonaitude 


Risk  Acceptable 


Inconclusive 

□ 


Risk  Unacceptable 


Fall  run,  FH  06,  FL  350  MSL.  Simulated  WRAP  MDT  turbulence  DI  with 

RT=0.9 


2007100200  FH:  006  Dl  BIAS:  I  WED  A  minus  WRAP  Tor  RT:  0.9  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


IWEDA  Optimistic  Bias  No  Bias  IWEDA  Pessimistic  Bias 


Fall  run,  FH  06,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 
WRAP  RT=0.9 


75 


b. 


Fall  Case:  FH  84 


By  84  hours  into  the  forecast  period,  the  ensemble  members  spread  out 
considerably,  revealing  considerably  more  uncertainty.  The  forecast  conveyed  by  the 
stochastic  product  (Figure  43b)  is  vastly  different  from  the  deterministic  (Figure  43a). 


2007100200  FH:  084  Deterministic  Forecast  (cOO)  MDT  or  greater  CAT  TURBC,  FL:  350  (MSL) 


2007100200  FH:  084  Forecast  Probability  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


-30  -20  -10  0  10  20  30  40 


Longitude 

Figure  43.  Fall  run,  FH  84,  FL  350  MSL.  a)  Deterministic  MDT  turbulence  forecast 
b)  Stochastic  MDT  turbulence  forecast  using  FP 


76 


As  time  evolved  in  the  forecast  period,  the  differences  between  IWEDA 
and  WRAP  became  larger  (Figure  44).  A  two  decision  input  bias  exists  in  area  1  (Figure 
44).  IWEDA  DI  deemed  unacceptable  risk  (Figure  45)  whereas  WRAP  DI  deemed 
acceptable  risk  (Figure  46).  Hence,  a  medium  risk  tolerant  user  was  given  a  no-go 
recommendation  from  IWEDA,  but  using  the  stochastic  tool  WRAP,  the  user  could  still 
operate  in  area  1 . 

In  area  2,  IWEDA  forecasted  green,  but  WRAP  forecasted  amber,  a  one 
decision  input  bias,  shown  by  the  optimistic  bias  (Figure  44).  Using  IWEDA,  the  medium 
risk  tolerant  user  would  be  given  a  false  sense  of  confidence  that  he/she  can  afford  any 
possible  impacts  from  MDT  turbulence  as  opposed  to  using  WRAP,  which  would 
communicate  the  possibility  of  intolerable  impacts. 


2007100200  FH:  004  DI  BIAS:  IWEDA  minus  WRAP  For  RT:  0.5  of  MDT  CAT  TURBO,  FL:  350  (MSL) 
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IWEDA  Optimistic  Bias  No  Bias  IWEDA  Pessimistic  Bias 


Figure  44.  Fall  run,  FH  84,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0.5 
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2007100200  FH:  084  Deterministic  (IWEDA-like)  Dl  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  45.  Fall  run,  FH  84,  FL  350  MSL.  Simulated  IWEDA  MDT  turbulence  DI 


2007100200  FH:  084  Stochastic  (WRAP-liko)  Dl  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  46.  Fall  run,  FH  84,  FL  350  MSL.  Simulated  WRAP  MDT  turbulence  DI  with 

RT=0.5 
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Similarly  to  FH  06,  the  I  WED  A  DI  bias  was  all  optimistic  with  WRAP’S 


RT=0.1,  but  the  differences  are  more  stark  (Figure  48).  IWEDA  was  green  in  area  3 
(Figure  45),  while  WRAP  communicated  unacceptable  risk  (red)  according  to  the  user’s 
RT  for  the  same  area  (Figure  47). 

Using  a  stochastic  stop-light  DI  tool  in  combination  with  the  FP  product 
can  help  the  education  process.  To  an  uneducated  user,  the  extent  of  unacceptable  risk  in 

Figure  47)  infers  an  abundance  of  MDT  or  greater  turbulence.  This  user 
can  look  at  the  FP  and  discover  many  areas  of  the  FP  are  highly  uncertain.  Using  the 
simple  concept  FP>RT,  the  user  can  understand  the  unacceptable  risk  areas. 


2007100200  FH:  084  Stochastic  (WRAP-like)  DI  forRT:  0.1  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  47.  Fall  run,  FH  84,  FL  350  MSL.  Simulated  WRAP  MDT  turbulence  DI  with 

RT=0.1 
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2007100200  FH:  084  Dl  BIAS:  IWEDA  minus  WRAP  for  RT:  0.1  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


IWEDA  Optimistic  Bias  No  Bias  IWEDA  Pessimistic  Bias 


Figure  48.  Fall  run,  FH  84,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0.1. 


Again,  the  DI  for  WRAP  changes  vastly  when  the  RT=0.9.  According  to 
the  WRAP  product,  all  users  can  operate  across  the  entire  AOR  (Figure  49)  while 
IWEDA  communicates  unacceptable  risk  across  the  northern  portions  of  the  AOR 
(Figure  45).  IWEDA  pessimistic  bias  is  evident  in  these  regions  (Figure  50).  Similar 
arguments  from  the  preceding  IWEDA- WRAP  comparison  apply  to  the  winter  case 
(Appendix  B). 


80 


2007100200  FH:  084  Stochastic  (WRAP-like)  Dl  forRT:  0.9  ofMDT  CAT  TURBC,  FL:  350  (MSL) 


Risk  Acceptable 
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Inconclusive  Risk  Unacceptable 

□  ■ 


Figure  49.  Fall  run,  FH  84,  FL  350  MSL.  Simulated  WRAP  MDT  turbulence  DI  with 

RT=0.9 


2007100200  FH:  084  Dl  BIAS:  IWEDA  minus  WRAP  for  RT:  0.9  ofMDT  CAT  TURBC,  FL:  350  (MSL) 


IWEDA  Optimistic  Bias  No  Bias  IWEDA  Pessimistic  Bias 


Figure  50.  Fall  run,  FH  84,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0.9. 
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3. 


Simulated  WRAP  versus  RSCH  Results 


Since  WRAP  and  the  RSCH  use  the  same  RT  and  FP  values,  the  maximum 
difference  is  only  one  decision  input  level.  It  is  not  possible  for  WRAP  to  forecast  green 
and  RSCH  to  forecast  red  and  visa-versa.  Differences  in  DI  stem  from  the  handling  of 
inconclusive  (amber)  regions,  as  described  in  the  methodology.  WRAP  compares  just  the 
FP  for  exceeding  the  marginal  threshold  with  the  RT  while  the  RSCH  method  considers 
the  uncertainty  of  FP  (ambiguity)  and  RT  by  comparing  FP  and  RT  confidence  intervals 
to  determine  the  decision  input.  To  aid  in  comparing  WRAP  and  RSCH  results,  the  worst 
case  potential  error  in  FP  is  plotted,  which  is  a  way  to  communicate  ambiguity.  The 
default  Cl  is  90%. 


a.  Fall  Case:  FH  06 

The  forecast  evolution  is  only  six  hours;  low  uncertainty  still  exists.  When 
compared  to  the  FP  product  (Figure  51a),  worst  case  potential  error  (PE)  areas  (Figure 
51b)  are  restricted  to  the  sharp  gradients  separating  the  zero  and  100%  probability.  Recall 
that  PE  decreases  toward  zero  and  as  FP  nears  extremes  (zero  and  100  percent),  so  there 
is  little  to  no  ambiguity  over  much  of  the  AOR.  The  most  PE  (%)  is  found  in  the  40-60 
where  uncertainty  is  at  its  maximum.  For  example,  point  A  has  FP  of  50%  where  the  PE 
is  14%.  Hence,  the  Cl  for  FP  at  this  point  can  range  36-64%. 


82 


-30  -20  -10  0  10  20  30  40 

Longitude 


b) 


2007100200  FH:  006  Worst  Case  Potential  Error  (+/-)  for  FP  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


-30  -20  -10  0  10  20  30  40 

Longitude 


Figure  51.  Fall  run,  FH  06,  FL  350  MSL.  a)  FP  of  MDT  turbulence  b)  Maximum 
(Worst  Case)  Potential  Error  for  FP  of  MDT  turbulence 
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The  case  in  Figure  52  displays  how  the  ambiguity  and  RT  intervals  used  in 
RSCH  can  alter  the  decision  input  to  the  user.  In  area  1,  WRAP  forecasted  green  (Figure 
53a),  but  RSCH  forecasted  amber  (Figure  53b).  Using  WRAP  DI,  a  user  would  appear  to 
have  affordable  impacts  from  MDT  turbulence  in  area  2,  but  using  the  RSCH,  the  DI  is 
inconclusive. 
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WRAP  Optimistic  Bias  No  Bias  WRAP  Pessimistic  Bias 


Figure  52.  Fall,  FH  06,  FL  350  MSL,  RT=0. 1 .  DI  Bias:  WRAP  minus  RSCH  chart. 


84 


a)  2007100200  FH:  006  Stochastic  (WRAP-like)  Dl  for  RT:  0.1  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  53.  Fall  run,  FH  06,  FL  350  MSL,  RT=0. 1  a)  Simulated  WRAP  MDT 
turbulence  DI  b)  RSCH  derived  MDT  turbulence  DI 
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In  the  case  displayed  in  Figure  54,  many  of  the  differences  (i.e.  areas  1 
and  2)  occur  because  WRAP  deemed  areas  inconclusive  (Figure  55a)  while  RSCH 
deemed  the  same  areas  acceptable  risk  (Figure  55b).  The  RT=0.5,  so  areas  greater  than 
50%  risk  of  exceeding  the  marginal  thresholds  for  the  WRAP  product  are  deemed 
inconclusive. 


2007100200  FH:  006  Dl  BIAS:  WRAP  minus  RSCH  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  54.  Fall,  FH  06,  FL  350  MSL,  RT=0.5.  DI  Bias:  WRAP  minus  RSCH  chart. 
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2007100200  FH:  006  Stochastic  (WRAP-like)  DI  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  55.  Fall  run,  FH  06,  FL  350  MSL,  RT=0.5  a)  Simulated  WRAP  MDT 
turbulence  DI  b)  RSCH  derived  MDT  turbulence  DI 
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If  RSCH  RT  uncertainty  interval  and/or  the  ambiguity  Cl  was  higher,  then 
areas  1-3  may  be  deemed  inconclusive  as  well.  The  differences  described  may  be  a  direct 
consequence  of  the  assumed  RT  uncertainty  interval  and  ambiguity  CL  Further  research 
is  needed  in  the  realm  of  decision  science  to  objectively  determined  and  test  the  RT 
interval.  To  explore  the  extent  of  the  affect  of  higher  ambiguity  CIs  on  the  WRAP  bias, 
the  researcher  plotted  the  RSCH  DI  and  DI  Bias  plots  using  0=0.95  and  0.99  (Figure 
56).  The  extent  of  amber  regions  in  areas  1-3  increased  slightly  by  incorporating  the 
higher  Cl  (Figure  56a, c),  but  the  majority  of  pessimistic  bias  still  exists  (Figure  56b, d). 
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2007100200  FH:  006  Research  Derived  DI  For  RT :  0.5  oFMDT  CAT  TURBC,  FL:  350  (MSL)  2007100200  FH:  006  DI  BIAS:  WRAP  minus  RSCH  for  RT :  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  56.  Fall,  FH  06,  FL  350  MSL,  RT=0.5.  CI=0.95:  a)  RSCH  DI  b)  DI  Bias: 
WRAP  minus  RSCH  chart;  CI=0.99:  c)  RSCH  DI  d)  DI  Bias:  WRAP  minus  RSCH  chart. 
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In  the  past  three  cases,  WRAP  DI  has  consistently  shown  a  pessimistic 
bias  trend  toward  the  RSCH  DI.  To  explain,  only  three  combinations  of  DI  can  constitute 
WRAP-RSCH  biases  (since  WRAP  and  RSCH  can  only  differ  by  one  DI).  The  first  two 
combinations  comprise  the  “type-one  differences,”  which  arise  from  the  inclusion  of 
ambiguity  and  RT  uncertainty  intervals  in  the  RSCH  DI.  WRAP  forecasted  red  (or 
green),  but  RSCH  forecasted  amber.  The  last  combination  of  DI  comprises  the  “type-two 
difference,”  which  arise  from  the  use  of  WRAP  marginal  threshold.  WRAP  forecasted 
amber,  but  RSCH  forecasted  green.  Biases  such  as  area  1  in  Figure  57  and  areas  1  and  2 
highlighted  in  Figure  54  are  examples  of  the  type-two  difference. 

The  DI  combination,  WRAP  =amber  and  RSCH  =  red,  for  the  type-two 
difference  is  not  possible.  For  example,  if  the  RT=0.3,  the  FP  of  TI=3.5  (marginal 
threshold)  must  exceed  30%  in  order  for  WRAP  to  forecast  amber.  If  the  FP  for 
exceeding  the  marginal  threshold  is  greater  than  30%,  then  the  FP  for  exceeding  the 
mission  threshold  must  be  less  than  30%.  Since,  the  RSCH  DI  uses  the  same  mission 
threshold,  the  worst  possible  decision  input  is  amber,  not  red. 

The  type-one  difference  has  both  an  optimistic  and  pessimistic  bias 
combination.  Since  it  is  not  possible  for  WRAP=amber  and  RSCH=red,  the  type-two 
difference  only  has  a  pessimistic  bias  combination.  Hence,  the  WRAP  DI  has  an  overall 
pessimistic  bias  towards  the  RSCH  DI.  If  the  marginal  threshold  was  slighter  higher,  say 
4.0  instead  of  3.5,  the  probability  of  exceeding  the  marginal  threshold  would  decrease 
and  pessimistic  bias  would  not  be  so  prevalent.  Consequently,  if  WRAP  used  no 
marginal  threshold,  the  amount  of  optimistic  and  pessimistic  biases  should  be  almost 
equal,  because  the  differences  would  be  driven  purely  by  the  type-one  differences. 
Therefore,  by  not  considering  the  uncertainty  of  the  FP  and  RT  and  using  a  marginal 
threshold,  WRAP  may  have  shifted  the  DI  to  a  degree  worse  than  it  might  actually  be. 
Users  may  be  unnecessarily  protecting  and/or  hesitating  when  only  portion  of  uncertainty 
information  is  included  into  the  DI. 
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2007100200  FH:  000  DI  BIAS:  WRAP  minus  RSCH  for RT:  0.9  of  MDT  CAT  TURBC,  FL:  300  (MSL) 
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Figure  57.  Fall,  FH  06,  FL  350  MSL,  RT=0.9.  DI  Bias:  WRAP  minus  RSCH  chart. 


b.  Fall  Case:  FH  84 

Much  uncertainty  evolved  as  time  progressed  to  FH  84,  reflected  by  the 
larger  areas  of  FP  40-60%  (Figure  58a)  and  the  dense  areas  of  high  PE  (Figure  58b).  In 
response,  the  RSCH  DI  calculated  more  areas  of  amber  and  larger  areas  of  difference 
between  WRAP  and  RSCH  DI  exist. 

The  RT=0.1  case  in  Figure  59  is  an  example  to  WRAP’S  type-one 
difference.  In  area  1,  the  FP  is  10-20%.  According  to  WRAP,  the  DI  is  unacceptable  risk 
Figure  60a).  The  maximum  PE  in  area  1  is  8%  which  makes  the  ambiguity  interval  2- 
18%.  The  RT  uncertainty  interval  is  8.3-11.7%.  Since  these  two  intervals  overlap,  the 
RSCH  DI  deemed  the  area  inconclusive  (Figure  60b). 
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b) 


Figure  58. 


2007100200  FH:  084  Forecast  Probability  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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2007100200  FH:  084  Worst  Case  Potential  Error  (+/-)  for  FP  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Fall  run,  FH  84,  FL  350  MSL.  a)  FP  of  MDT  turbulence  b)  Maximum 
(Worst  Case)  Potential  Error  for  FP  of  MDT  turbulence 
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Figure  59.  Fall,  FH  84,  FL  350  MSL,  RT=0. 1 .  DI  Bias:  WRAP  minus  RSCH  chart. 
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Figure  60.  Fall  run,  FH  84,  FL  350  MSL,  RT=0. 1  a)  Simulated  WRAP  MDT 
turbulence  DI  b)  RSCH  derived  MDT  turbulence  DI 
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Figure  60  continued. 


The  case  in  Figure  61  is  a  rare  balance  of  optimistic  and  pessimistic  biases 
because  the  majority  of  the  differences  are  type-one  differences.  In  area  1,  the  maximum 
FP  for  MDT  turbulence  in  area  1  is  60%,  which  WRAP  deemed  as  unacceptable  risk.  The 
overlap  of  the  ambiguity  intervals  (48-72%)  the  RT  uncertainty  intervals  (45-55%)  make 
the  RSCH  DI  inconclusive.  In  this  instance,  WRAP  may  have  shut  down  operations 
unnecessarily.  In  area  2,  the  maximum  FP  for  MDT  is  just  below  50%,  hence  WRAP 
deemed  the  area  acceptable  risk.  The  overlap  of  the  ambiguity  intervals  (35-63)  with  the 
RT  uncertainty  intervals  make  the  RSCH  DI  inconclusive  once  again.  In  this  instance, 
WRAP  may  have  left  the  user  vulnerable  to  a  possible  threat  because  the  DI  is  too  close 
to  call. 
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2007100200  FH:  084  Dl  BIAS:  WRAP  minu3  RSCH  For  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  61.  Fall,  FH  84,  FL  350  MSL,  RT=0.5.  DI  Bias:  WRAP  minus  RSCH  chart. 


Minimal  differences  exists  between  the  WRAP  and  RSCH  DI  products  in 
the  case  displayed  in  Figure  64.  The  shape  of  positive  definite  PDF  drives  this  difference. 
In  this  highly  uncertain  situation,  the  positive  definite  TI  PDF  is  spread  out  (for  example, 
Figure  62).  From  Figure  62,  with  RT=0.9  and  FP  for  exceeding  the  marginal 
threshold=90.5%,  WRAP  forecasts  amber.  RSCH,  with  its  overlapping  ambiguity  and  RT 
uncertainty  intervals,  also  forecasts  amber. 


Figure  62.  Positive  Definite  TI  PDF  with  high  probability  of  high  TI 
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So  the  majority  of  the  DI  for  both  WRAP  and  RSCH  are  equal.  Little 
value  is  added  by  incorporating  ambiguity  and  RT  uncertainty  information  when  the 
forecast  has  high  uncertainty  and  the  RT  is  also  high.  In  the  case  associated  with  Figure 
59,  the  majority  of  the  biases  are  in  areas  of  FP  ranging  from  10-30%.  The  FP  for 
mission  threshold  >  RT,  so  WRAP  forecasted  red.  The  overlapping  intervals  for  RSCH 
cause  the  DI=amber  (Figure  63).  The  differences  are  much  more  prevalent  at  the  low  RT. 
Hence,  in  highly  uncertainty  weather  situations,  users  with  low  RT  such  as  those 
accomplishing  training  would  benefit  more  using  the  RSCH  DI  than  users  with  a  high 
RT,  such  as  those  accomplishing  mission  critical  operations.  Similar  arguments  from  the 
preceding  WRAP-RSCH  comparison  apply  to  the  winter  case  (Appendix  C). 


Marginal  Threshold:  TI=3.5 
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Figure  63.  Positive  Definite  TI  PDF  with  high  probability  of  low  TI 
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2007100200  FH:  084  Dl  BIAS:  WRAP  minus  RSCH  forRT:  0.9  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  64.  Fall,  FH  84,  FL  350  MSL,  RT=0.9.  DI  Bias:  WRAP  minus  RSCH  chart. 
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V.  CONCLUSIONS 


A.  FINAL  REMARKS 

The  three  main  objectives  of  the  thesis  were  to:  1)  furnish  a  process  to  account  for 
uncertainty  in  both  the  ensemble  data  and  user  risk  tolerance  for  the  decision  input,  2) 
create  an  effective  visualization  to  for  varying  levels  of  risk  intolerance,  mission 
thresholds  for  turbulence,  flight  levels,  and  forecast  hours  that  is  educational  and  practical 
for  the  user  3)  demonstrate  potential  value  added  to  the  user  when  using  the  research 
derived  decision  input  product  through  a  comparison  of  current  decision  input  products. 

Chapter  II  described  the  background  to  atmospheric  prediction,  ensemble 
forecasting  (including  the  ambiguity),  weather  risk  management,  and  the 
calculation/communication  of  forecast  uncertainty  (from  today’s  decision  input  tools). 
Using  the  background  as  a  foundation,  Chapter  III  laid  out  a  three  part  process  to 
calculate  and  produce  visualization  products  for  calibrated  forecast  probability,  worst 
case  potential  error,  and  the  optimized  decision  input.  Through  the  Taijitu  method,  the 
forecast  probability  was  calibrated  and  the  worst  case  potential  error  was  calculated.  The 
decision  input  used  objective  estimates  of  the  ambiguity  intervals  and  subjective 
estimates  of  the  risk  tolerance  uncertainty  intervals.  A  prototype  GUI  was  constructed  so 
a  non-educated  user  can  easily  access  these  products  according  to  his/her  unique  RT, 
mission  threshold  of  turbulence,  flight  level,  and  forecast  hour.  Chapter  IV  fulfilled  the 
last  objective  with  two  cases  which  compared  current  decision  input  products  with  the 
research  results  using  scenarios  of  different  risk  tolerances  and  forecast  hours. 

Using  the  ninety  percent  confidence  interval  for  ambiguity  communicated  a  large 
amount  of  inconclusiveness  but  still  yielded  definitive  decision  inputs.  The  maximum 
bounds  for  the  ambiguity  increased  with  increased  CIs,  but  remained  nearly  constant 
through  the  forecast  period,  since  the  constant  error  variance  of  the  ensemble  mean 
appears  to  have  dominated  the  ambiguity.  As  forecast  period  evolved,  the  average  error 
in  the  ensemble  spread  varied  due  to  weak  ensemble  dispersion  and  excessive  initial 
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conditions  perturbations.  In  response,  the  calibration  curve  reversed  due  to  the  change  in 
average  ensemble  spread  error  from  >1  to  <1.  Incorporating  uncertainty  of  FP  and  RT 
into  the  DI  process  can  often  alter  the  decision  input.  The  differences  between  of  the 
different  decision  tools  show  potential  value  added  to  the  user  when  FP  and  RT 
uncertainty  information  is  incorporated. 

The  differences  shown  in  the  comparisons  of  the  current  decision  tools  showed 
potential  value  added  to  the  decision  making  process  when  increasing  degrees  of 
uncertainty  information  is  included  into  the  DI.  IWEDA  is  a  generic  decision  aid  made  to 
give  DI  to  all  users  with  one  product  but  is  limited  by  using  only  one  forecast  solution 
and  does  not  consider  RT.  WRAP’S  inclusion  of  stochastic  weather  information  and  RT 
generates  a  unique  DI  which  can  benefit  user  in  the  long  term  by  minimizing  their 
expenses.  The  results  confirmed  the  IWEDA  DI  bias  is  highly  dependant  on  RT  and 
degree  of  uncertainty.  WRAP  DI  changes  with  different  RT  inputs.  Increasing  the  RT 
gave  more  acceptable  risk  DIs  because  of  the  users  ability  to  afford  potential  loss  (high 
C/L)  and  decreasing  the  RT  gave  more  unacceptable  risk  because  of  the  user  inability  to 
afford  the  potential  loss  (low  C/L).  Compared  with  WRAP  using  a  low  (high)  RT, 
IWEDA  (the  deterministic  DI),  had  an  optimistic  (pessimistic)  bias.  As  the  forecast 
evolution  progresses,  the  uncertainty  increased  and  the  areas  of  bias  grew. 

By  comparing  the  error  bars  of  FP  and  RT  to  create  the  DI,  the  RSCH  product 
sometimes  shifted  the  DI  from  WRAP.  Overall,  WRAP  had  a  pessimistic  bias  relative  to 
the  RSCH  DI.  Only  three  combinations  of  DI  can  constitute  WRAP-RSCH  differences 
(since  WRAP  and  RSCH  can  only  differ  by  one  DI).  The  first  two  combinations 
comprise  the  “type-one  differences,”  which  arise  from  the  inclusion  of  ambiguity  and  RT 
uncertainty  intervals  in  the  RSCH  DI.  Type-one  differences  consist  of  both  pessimistic 
and  optimistic  differences.  The  last  combination  of  DI  comprises  the  “type-two 
difference,”  which  arise  from  the  use  of  WRAP  marginal  threshold.  Type-two  difference 
consists  of  only  a  pessimistic  difference.  The  DI  combination,  WRAP  =  amber  and 
RSCH  =  red,  for  the  type-two  difference  is  not  possible  due  to  WRAP’S  marginal 
threshold.  Therefore,  an  imbalance  of  pessimistic  bias  occurred.  Increasing  the  Cl  for  the 
ambiguity  intervals  only  decreased  WRAPs  pessimistic  bias  slightly.  Therefore,  by  not 
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considering  the  uncertainty  of  the  FP  and  RT  and  using  a  marginal  threshold,  WRAP  may 
have  shifted  the  DI  to  a  degree  worse  than  it  might  actually  be,  preventing  users  from 
taking  action.  If  the  differences  between  WRAP  and  RSCH  were  only  due  to  type-one 
differences,  then  there  would  be  no  overall  bias. 

WRAP-RSCH  differences  were  few  early  in  the  forecast  period  because  of  the 
low  uncertainty.  As  uncertainty  increased,  so  did  the  ambiguity.  The  DI  differences  grew 
in  spatial  coverage  because  of  the  differences  in  calculating  the  inconclusive  (amber) 
regions.  In  high  uncertainty  situations,  the  RSCH  methods  are  potentially  valuable  in  the 
low  to  middle  RT  range  (i.e.  training  and  some  operational  mission)  but  not  high  RT 
range  (i.e.  mission  critical  operations)  due  to  the  shape  of  the  TI  PDF. 

The  study  explored  an  evolution  of  forecast  honesty:  from  a  deterministic  DI  that 
communicates  no  forecast  uncertainty  to  a  stochastic  DI  that  considers  forecast 
uncertainty  (FP)  and  RT  to  another  stochastic  DI  that  considers  uncertainty  of  both  FP 
and  RT.  The  differences  in  DI  can  affect  the  users’  actions.  The  optimized  decision  input 
derived  from  this  research  incorporates  a  complete  portrait  of  uncertainty  so  that  the  war¬ 
fighter  can  make  the  most  cost  effective  decision.  The  inconclusive  (amber)  DI  gives  the 
decision  maker  the  option  to  go/not-go  based  on  other  decision  factors  that  may  not  be 
quantifiable  like  morale,  importance  of  mission,  and  personnel  safety.  The  research  has 
demonstrated  effective  communication  and  integration  of  optimized  decision  input  into 
the  decision  making  process.  Procedures  and  results  of  research  can  serve  as  an  example 
to  further  education  and  development  of  stochastic  methods  in  the  Air  Force  and 
Department  of  Defense  to  help  improve  combat  capability. 

B.  FURTHER  RESEARCH  AND  RECOMMENDATIONS 

1.  Further  Research 

Although  the  Taijitu  method  introduced  in  this  research  uses  sound  mathematical 
and  statistical  methods,  a  study  to  verify  and  validate  the  method  is  needed. 
Additionally,  decision  science  research  needs  to  be  accomplished  to  determine  an 
objective  RT  uncertainty  interval  for  military  operations.  Verification  of  the  ensemble 
data  was  done  for  one  season,  but  needs  to  be  done  for  all  four  seasons  in  order  to 
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accurately  reflect  the  breadth  of  error  in  the  ensemble  mean  and.  To  ascertain  how  the 
level  of  understanding  of  DI  product  attributes  enhance  or  diminish  the  usability  of  the 
visualization,  an  objective  process  should  be  devised  to  solicit  feedback  from  various 
operational  users  such  as  pilots.  To  quantify  combat  capability  improvement,  a  robust 
case  study  using  an  operational  units’  data  should  be  accomplished. 

2.  Recommendations 

Results  show  potential  value  added  if  current  DoD  decision  aid  products  include 
ambiguity  and  RT  uncertainty.  In  order  to  help  the  transition  to  the  stochastic  mindset, 
user-friendly  products  need  to  made  available  so  that  user  can  easily  integrate  the 
information  into  the  decision  making  process.  A  possible  method  is  an  online  GUI 
capability  made  available  at  the  strategic  level,  so  regional  weather  centers  (Operational 
Weather  Squadrons)  and  individual  weather  units  (Combat  Weather  Teams)  can  access 
the  stochastic  weather  information  and  calculate  DI  for  their  user.  The  resultant  GUI 
product  could  be  a  three  panel  plot  showing  the  calibrated  FP,  worst-case  potential  error, 
and  stop-light  decision  input  for  one  weather  parameter.  To  show  multiple  parameters,  a 
decision  input  matrix  could  be  constructed  according  to  the  unit’s  requirements.  The 
matrix  would  include  the  mission  thresholds,  RT  for  each  threshold,  FP  of  exceeding  the 
threshold,  and  the  DI.  The  highest  (worst-case)  recommendation  will  be  the  single  input, 
overall  recommendation  for  the  mission  (Table  7).  The  mission  DI  matrix  could  easily  be 
formulated  automatically  from  machine-to-machine  inputs. 
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Mission  Decision  Input  (DI)  Matrix 

Mission  Threshold 

SVR 

Turbulence 

MDT 

Rime 

Icing 

15  knots 
Crosswinds 

1  Statue  Mile 
Visibility 

Risk  Tolerance/ 
Uncertainty  Interval 

10% 

±  1.7% 

30% 

±4% 

20% 

±3% 

40% 

±  4.5% 

Risk  (FP)/ 
Ambiguity  Interval 

0% 

±0% 

5% 

±  0.5% 

28% 

±3% 

35% 

±4% 

DI 

Acceptable 

Risk 

Acceptable 

Risk 

Inconclusive 

Overall  Mission  DI/ 
Recommendation 

Table  7.  Prototype  Mission  Decision  Input  Matrix 
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APPENDIX  A:  RESEARCH  DI  WITH  DIFFERENT  Cl  OF 

AMBIGUITY 


2007100200  FH:  024  Decicision  Input  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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2007100200  FH:  024  Decicision  Input  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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2007100200  FH:  024  Decicision  Input  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  65.  (a-e)  RSCH  DI  for  MDT  turbulence  using  a  range  of  CIs;  fall  run,  FH  24, 

FL  350  (MSL),  RT=0.5 
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2007100200  FH:  040  Decicision  Input  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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FH:  040  Decicision  Input  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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2007100200  FH:  040  Decicision  Input  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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2007100200  FH:  040  Decicision  Input  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  66.  (a-e)  RSCH  DI  for  MDT  turbulence  using  a  range  of  CIs;  fall  run,  FH  48, 

FL  350  (MSL),  RT=0.5 
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2007100200  FH:  084  Decicision  Input  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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2007100200  FH:  084  Decicision  Input  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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2007100200  FH:  084  Decicision  Input  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


Longitude 


Figure  67.  (a-e)  RSCH  DI  for  MDT  turbulence  using  a  range  of  CIs;  fall  run,  FH  84, 

FL  350  (MSL),  RT=0.5 
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APPENDIX  B:  WINTER  CASE:  SIMULATED  IWEDA  VERSUS 

SIMULATED  WRAP 


The  following  are  figures  from  the  simulated  IWEDA  and  WRAP  comparisons  of 
January  30,  2008,  00Z  (winter)  run.  For  reasoning,  see  similar  cases  in  Chapter  III, 
section  B.2. 

A.  FORECAST  HOUR:  06 


2008013000  FH:  006  Deterministic  Forecast  (cOO)  MDT  or  greater  CAT  TURBC,  FL:  350  (MSL) 


Longitude 


-30  -20  -10  0  10  20  30  40 

Longitude 


Figure  68.  Winter  run,  FH  06,  FL  350  MSL.  a)  Deterministic  MDT  turbulence 
forecast  b)  Stochastic  MDT  turbulence  forecast  using  FP 
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2008013000  FH:  006  Stochastic  (WRAP-liko)  Dl  forRT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Inconclusive 
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Risk  Unacceptable 


Figure  69.  Winter  run,  FH  06,  FL  350  MSL.  a)  Simulated  IWEDA  MDT  turbulence 
DI  b)  Simulated  WRAP  MDT  turbulence  DI  with  RT=0.5 
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2008013000  FH:  000  Dl  BIAS:  I  WED  A  minus  WRAP  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  70.  Winter  run,  FH  06,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0.5. 
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2008013000  FH:  000  Stochastic  (WRAP-like)  Dl  for  RT:  0.1  of  MDT  CAT  TURBO,  FL:  350  (MSL) 


Longitude 


Risk  Acceptable 


Inconclusive  Risk  Unacceptable 

□  ■ 


Figure  71.  Winter  run,  FH  06,  FL  350  MSL.  a)  Simulated  IWEDA  MDT  turbulence 
DI  b)  Simulated  WRAP  MDT  turbulence  DI  with  RT=0.1 
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2008013000  FH:  006  Dl  BIAS:  IWEDA  minus  WRAP  for  RT:  0.1  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  72.  Winter  run,  FH  06,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0.1. 
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2008013000  FH:  006  Stochastic  (WRAP-like)  Dl  for  RT:  0.9  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


Longitude 


Risk  Acceptable 


Inconclusive 
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Risk  Unacceptable 


YFigure  73.  Winter  run,  FH  06,  FL  350  MSL.  a)  Simulated  IWEDA  MDT  turbulence 
DI  b)  Simulated  WRAP  MDT  turbulence  DI  with  RT=0.9 
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Figure  74.  Winter  run,  FH  06,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0.9. 
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B.  FORECAST  HOUR:  84 


200801 3000  FH:  084  Deterministic  Forecast  (cOO)  MDT  or  greater  CAT  TURBC,  FL:  350  (MSL) 


Longitude 

Figure  75.  Winter  run,  FH  84,  FL  350  MSL.  a)  Deterministic  MDT  turbulence 
forecast  b)  Stochastic  MDT  turbulence  forecast  using  FP 
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2008013000  FH:  084  Deterministic  (IWEDA-like)  Dl  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


Figure  76. 


Longitude 


2008013000  FH:  084  Stochastic  (WRAP-like)  Dl  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 


Longitude 


Risk  Acceptable 


Inconclusive 
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Risk  Unacceptable 


Winter  run,  FH  84,  FL  350  MSL.  a)  Simulated  IWEDA  MDT  turbulence 
Dl  b)  Simulated  WRAP  MDT  turbulence  Dl  with  RT=0.5 
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2008013000  FH:  084  Dl  BIAS:  IWEDA  minus  WRAP  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  77.  Winter  run,  FH  84,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0.5. 
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2008013000  FH:  084  Deterministic  (IWEDA-like)  Dl  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  78.  Winter  run,  FH  84,  FL  350  MSL.  a)  Simulated  IWEDA  MDT  turbulence 
Dl  b)  Simulated  WRAP  MDT  turbulence  Dl  with  RT=0.1 
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Latrtude 


2008013000  FH:  084  Dl  BIAS:  I  WED  A  minus  WRAP  for  RT:  0.1  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  79.  Winter  run,  FH  84,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0.1. 
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2008013000  FH:  084  Stochastic  (WRAP-like)  Dl  forRT:  0.9  oFMDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  80.  Winter  run,  FH  84,  FL  350  MSL.  a)  Simulated  IWEDA  MDT  turbulence 
DI  b)  Simulated  WRAP  MDT  turbulence  DI  with  RT=0.9 
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2008013000  FH:  084  Dl  BIAS:  IWEDA  minus  WRAP  for  RT:  0.9  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  81.  Winter  run,  FH  84,  FL  350  MSL.  DI  Bias:  IWEDA  minus  WRAP  chart. 

WRAP  RT=0.9. 
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APPENDIX  C:  WINTER  CASE:  SIMULATED  WRAP  VERSUS 

RSCH  RESULTS 


The  following  are  figures  from  the  simulated  WRAP  and  research  results  (RSCH) 
comparisons  of  January  30,  2008,  00Z  (winter)  run.  For  reasoning,  see  similar  cases  in 
Chapter  III,  section  B.3. 

A.  FORECAST  HOUR:  06 


2008013000  FH:  006  Forecast  Probability  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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2008013000  FH:  006  Worst  Case  Potential  Error  (+/-)  for  FP  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  82.  Winter  run,  FH  06,  FL  350  MSL.  a)  FP  of  MDT  turbulence  b)  Maximum 
(Worst  Case)  Potential  Error  for  FP  of  MDT  turbulence 
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a) 


2008013000  FH:  008  Stochastic  (WRAP-like)  Dl  for  RT:  0.1  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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2008013000  FH:  008  Research  Derived  Dl  ForRT:  0.1  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  83.  Winter  run,  FH  06,  FL  350  MSL,  RT=0. 1  a)  Simulated  WRAP  MDT 

turbulence  Dl  b)  RSCH  derived  MDT  turbulence  Dl 


122 


2008013000  FH:  006  Dl  BIAS:  WRAP  minus  RSCH  for  RT:  0.1  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  84.  Winter  run,  FH  06,  FL  350  MSL,  RT=0. 1 .  DI  Bias:  WRAP  minus  RSCH 

chart. 
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Figure  85.  Winter  run,  FH  06,  FL  350  MSL,  RT=0.5  a)  Simulated  WRAP  MDT 
turbulence  DI  b)  RSCH  derived  MDT  turbulence  DI 
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2008013000  FH:  006  Dl  BIAS:  WRAP  minus  RSCH  for  RT:  0.5  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  86.  Winter  run,  FH  06,  FL  350  MSL,  RT=0.5.  DI  Bias:  WRAP  minus  RSCH 

chart. 
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Figure  87.  Winter  run,  FH  06,  FL  350  MSL,  RT=0.9  a)  Simulated  WRAP  MDT 
turbulence  DI  b)  RSCH  derived  MDT  turbulence  DI 
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2008013000  FH:  006  Dl  BIAS:  WRAP  minus  RSCH  for  RT:  0.9  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  88.  Winter  run,  FH  06,  FL  350  MSL,  RT=0.9.  DI  Bias:  WRAP  minus  RSCH 

chart. 
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Figure  89.  Winter  run,  FH  84,  FL  350  MSL.  a)  FP  of  MDT  turbulence  b)  Maximum 
(Worst  Case)  Potential  Error  for  FP  of  MDT  turbulence 
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Figure  90.  Winter  run,  FH  84,  FL  350  MSL,  RT=0. 1  a)  Simulated  WRAP  MDT 

turbulence  DI  b)  RSCH  derived  MDT  turbulence  DI 
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Figure  9 1 .  Winter  run,  FH  84,  FL  350  MSL,  RT=0. 1 .  DI  Bias:  WRAP  minus  RSCH 

chart. 
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Figure  92.  Winter  run,  FH  84,  FL  350  MSL,  RT=0.5  a)  Simulated  WRAP  MDT 
turbulence  Dl  b)  RSCH  derived  MDT  turbulence  Dl 
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Figure  93.  Winter  run,  FH  84,  FL  350  MSL,  RT=0.5.  DI  Bias:  WRAP  minus  RSCH 

chart. 
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Figure  94.  Winter  run,  FH  84,  FL  350  MSL,  RT=0.9  a)  Simulated  WRAP  MDT 
turbulence  DI  b)  RSCH  derived  MDT  turbulence  DI 
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2008013000  FH:  084  Dl  BIAS:  WRAP  minus  RSCH  for  RT:  0.9  of  MDT  CAT  TURBC,  FL:  350  (MSL) 
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Figure  95.  Winter  run,  FH  84,  FL  350  MSL,  RT=0.9.  DI  Bias:  WRAP  minus  RSCH 

chart. 
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